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Chapter 1 

What is a Stochastic Process? 



The word stochastic is jargon for random. A stochastic process is a system 
which evolves in time while undergoing chance fluctuations. We can 
describe such a system by defining a family of random variables, {X,}, 
where X t measures, at time t, the aspect of the system which is of interest. 
For example, X t might be the number of customers in a queue at time t. 
As time passes, customers will arrive and leave, and so the value of X t will 
change. At any time t, X t takes one of the values 0, 1,2,...: and t can be 
any value in a subset of (— 00 , 00 ), the infinite past to the infinite future. 
If we observe the queue continuously, and customers arrive one at a time 
to be served by a single server, then, when a customer arrives, the value 
of X t , the queue size, increases by one, and when a customer departs after 
being served, X, decreases by one (Figure 1.1). The values which X t 



Figure l.l The number of customers, X„ in a queue at time t 

can take are called its states, and changes in the value of X t are called 
transitions between its states. If we observe the queue size not continu¬ 
ously but at unit intervals, say once every quarter of an hour, then more 
than one customer can arrive or leave in each time interval. This will lead 
to larger fluctuations in the value of X r These obvious statements repre¬ 
sent the basis of the construction of a model of a queue which incorporates 
random intervals between the arrivals of customers and random periods 
spent at the service point. It is the often complex consequences of these 
idealised models which we shall be studying. Simple as these models are, 
by incorporating a little of the randomness of the real world they bring 
us far closer to understanding the real world than could possibly be 
achieved with models which neglect such random behaviour. 

Stochastic models are applicable to any system involving chance vari¬ 
ability as time passes. In geophysics they have been used for the prediction 
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of the size and whereabouts of earthquakes, in geography to study the 
spread of shoe shops in a growing city, in entomology the way in which 
aphids congregate on the leaves of plants, in nature conservancy the way 
in which birds, turtles and eels navigate, and in industry they have been 
used for the prediction of the durations of strikes. 

By a stochastic process , we shall mean a family of random variables 
{ X t }, where t is a point in a space T called the parameter space , and where, 
for each t e T, X t is a point in a space S called the state space. 

The family {X f } may be thought of as the path of a particle moving 
‘randomly’ in space S, its position at time t being X r A record of one of 
these paths is called a realisation of the process. 

We are interested in relations between the X t for different fixed values of 
t. We apply the theory of probability to determine these relationships. 

Other aspects of stochastic processes will also interest us. For example, 
the time which elapses before a gambler loses all his capital, or the chance 
that any customer who might enter a shop in a fixed interval (0, t) will be 
served without having to wait. By thinking in terms of a particle travelling 
in a space we can often demonstrate the applicability of the same model 
to widely differing situations. For example, the time to a gambler’s ruin, 
the time before a reservoir dries up, the time for the server of a queue to 
become free and the time before an animal population becomes extinct are 
all equivalent to the time before the particle first hits a point O. At that 
time the gambler’s resources have been reduced to zero, the water level 
is down to zero, the number of waiting customers is zero and the population 
size is zero. 

Problem 1.1 What are the state space and parameter space for a stochastic 
process which is the score during a football match? 

Solution. The state space S is the set of possible values the score can take, 
so S = {(x,y): x,y = 0,1,2,...}. If we measure time in minutes, then the 
parameter space T is (0,90). The process starts in state (0,0), and transi¬ 
tions take place between the states of S whenever a goal is scored. A goal 
increases x or y by one, so the score (x, y) will then go to (x+l,y) or 
(x,y+l). □ 

Problem 1.2 Describe how we might use a stochastic process to study 
the resources of an insurance company. 

Solution. We let the resources at time t be a random variable X t . Then 
X t will increase at a randomly fluctuating but fairly steady rate as premiums 
come in, but is subject to sudden falls as claims are met. □ 
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Problem 1.3 What are the state space and parameter space for a 
stochastic process which is the depth of the sea in position x at time r? 

Solution. The depth of the sea is measured from the top of a wave 
down to the seabed. As the waves move about, the depth at any fixed 
point x will vary with time, regardless of any larger-scale influences such 
as tides. We can measure the depth at any time t, and in any position x, 
so the parameter space Tis the set of all t = (t, x) for which — oo < t < co 
and xeJ, where X is the set of map references for the entire sea. Here t 
is not just time, but a combination of time and space coordinates. The 
state space S is the set of all values which the depth can possibly be, so 
S = [0, oo ), where the depth is 0 when the seabed is exposed, and we do 
not limit the height the waves can reach, although a wave of infinite height 
will not occur without a miracle. □ 

Problem 1.4 An epidemic process. A fatal disease is brought into a 
closed community by a single infected individual. Describe how the 
spread of the disease may be studied as a stochastic process. 

Solution. We suppose that, for a period, infected persons show no 
symptoms and are not infectious. They then become carriers, and are 
infectious, but still show no symptoms. Carriers, after a period, exhibit 
symptoms of the disease and are isolated These people are cured and be¬ 
come immune, or else they die. 

Let M be the class of immune members of the community; S be the class 
of susceptibles, that is, people at risk; N be the noninfectious incubators 
of the disease; C, the carriers; /, the carriers who have been isolated: and 
Z), the dead. Initially all members of the community are in M or S, except 
the single infected person in N or C. Transitions between classes take 
place only according to the arrows in Figure 1.2. 

The random variables of interest are the numbers in each of the classes 
at each time t. The progress of the disease will depend on the degree of 



Figure 1.2 
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immunity, the amount of contact between carriers and susceptibles, the 
rate at which carriers are detected, and the chance of a cure being effected. 
Epidemiologists use the theory of stochastic processes to seek ways of 
influencing the rates of transition between the classes. □ 

Problem 1.5 A library book loan process. A reader visits a library 
regularly at the same time each week. There, if he has finished the book 
he is currently borrowing, he exchanges it; otherwise he has its loan 
renewed. Consider the stochastic process {Z„: n = .... —1,0,1,2,...}, 
where Z„ is the number of renewals of the book currently being borrowed 
as the reader leaves the library in week n, where the weeks are measured 
from an arbitrary time point. 

If a book has just been exchanged, then Z„ = 0. What is the state space 
for this process, and what are the possible transitions of state? 

Solution. State space S is {0,1,2,...}, since Z„ can take any value 
0, 1 , 2 ,...; though it would have to be a massive tome for very large values. 
If = k(k = 0,1,2,...), then in week n + 1 if the book being read is com¬ 
pleted, it is exchanged and a new book is borrowed, so Z„ + , =0. Otherwise, 
the loan is renewed for another week, so Z n+1 = k+ 1. □ 

Problem 1.6 A dam storage process. Consider a dam which can hold at 
most w units of water. Suppose that during day n, y n units of water flow 
into the dam, any overflow being lost. Provided the dam is not dry, one 
unit of water is released at the end of each day. Suppose that {y n } is a 
sequence of nonnegative integers, and that w is a positive integer. What is 
the state space for {Z„: n = 0,1,2,...}, where Z n is the content of the dam 
after the release (if any) of the unit of water on day n? Show how Z B + 
depends on Z n and y n+ ,. 

Solution. If the dam is full on day n, then after the unit release, Z n will 
take value w-1. If we then have a dry day (i.e. y n+1 = 0), Z n+ , = w- 2. 
After another dry day (y n+2 = 0),Z„ +2 = w-3; and so on, until after the 
(w—l)st dry day, Z n+W _ 1 = 0. Clearly these will be the only possible 
values that Z H can take. The state space is therefore {0,1,2,...,w -1}. 

If Z„ = i ( i = 0, 1,.... w 1) and y H+l = k(k = 0,1, 2,...), then 


Z-i = 


0 

i+k- 1 
yv— 1 


(i+k = 0 or 1) 

(i+k = 2,3,...,w-1) 
(i+k = w,w+l,...) 


□ 


Problem 1.7 A factory has two machines, but on any given day not 
more than one is in use. This machine has a constant probability p of 
breaking down and, if it does, the breakdown occurs at the end of the 
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day’s work. A single repairman is employed. It takes him two days to 
repair a machine, and he works on only one machine at a time. Construct 
a stochastic process which will describe the working of this factory. 

Solution. We must select a suitable random variable to observe. Clearly 
its value need be recorded only at the end of each day, since all transitions 
occur just before then. The parameter space T will therefore be the set of 
working days during which this system is in use. Let us call the first day, 1; 
the second, 2; and so on; then T = {1,2,3,...}. 

A suitable stochastic process is {X n :neT}, where we record at the 
end of day n the value of X n , the number of days that would be needed to 
get both machines back in working order. If both machines are in working 
order, then X n is 0. If one machine is in working order and the other has 
already had one day’s repair carried out on it, then X n is 1. If one machine 
is in working order, and the other has just broken down, then X n is 2. 
If one machine has just broken down, and the other has had one day’s 
repair carried out on it, then X n is 3. These are the only possible cases, so 
the state space is S = {0,1,2,3}. □ 

EXERCISES 

1. List the set of possible transitions between the states of the factory 
described in Problem 1.7. 

2. Choose random variables suitable for studying the behaviour of 
traffic at the junction between a main road and a side road. 

3. Construct a process for studying the counting of the votes in an elec¬ 
tion fight between just two candidates. 

4. Describe the process {X,}, where X t is the number of teeth an indi¬ 
vidual has at time t if he was born at time t = 0. Clearly X 0 = 0. 
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Chapter 2 

Results from Probability Theory 

This chapter contains results from elementary probability theory. Those 
to whom this is all familiar should nevertheless read it to acquaint them¬ 
selves with the notation, some of which has been introduced specifically 
to facilitate the study of stochastic processes, and so differs from that 
generally used. 

2.1 Introduction to probability theory An experiment is any situation 
in which there is a set of possible outcomes. For example, a competition 
or game, a horse race, a ballot, a law suit are all experiments since their 
results are uncertain. A random variable (which we abbreviate to rv) is a 
number associated with the outcome of an experiment. For example, 
the experiment might be for a dentist to observe the number of natural 
teeth a patient has. Then the rv, X. will take value 28 if the patient has 
28 teeth. An rv is called discrete if it can take only a finite or countably 
infinite number of distinct values. For example, the number of teeth 
must be one of the numbers 0 to 32. 

Since the outcomes of an experiment are uncertain, associated with each 
outcome there will be a probability. The probability that rv X takes value 
x will be the probability that the outcome associated with x occurs. 
We write this: pr(X = x). If a discrete rv, X, can take values x x . x 2 ,... 
in a set S, then the sequence of numbers P XI ,P X ,,..., where 

p x = pr(X = x) (x 6 S) 

is called the probability distribution of the rv X. and 

P x > 0(xe S), I p x = p H-p +... = I 

xeS 

For any subset S* of S 

Pr (Xe S*) = £ P x (2.1) 

xeS* 

For example, in throwing a die, if X is the value it shows, then S = {1,2,3, 
4,5,6}. If the die is fair then p x = % (x e S). To find the probability that 
an even number shows, we define S* = {2,4,6}, then 

pr(an even number shows) = pr(X e S*) = pr( V = 2 or 4 or 6) 

= Pi+Pt+Pf, 

If pr(X = x) = 0 for every real x, then rv X is called continuous. For 
example, X might be a height or a weight. Its value will have an infinite 
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decimal expansion, although in practice we must round off our measure¬ 
ments. 

The distribution function (abbreviation: df) of an rv X is the function 
F(x) = pr(X x) 

F is a nondecreasing function of x, and 

F(-oo) = 0, F(oo) = 1 

If there is a nonnegative function /(x) such that we can write 


then 


F(x) =j _ x f{y)dy 


/(*) = 


dF(x) 

dx 


is the probability density function (abbreviation: density) of X, and X is 
continuous. For example, if rv X has an exponential distribution with 
parameter A, then 


F(x) = 



(x < 0) 
(x > 0)’ 



(x 0) 
(x > 0) 


( 2 . 2 ) 


We write: X is <?(/). 

We can interpret the density as a probability by writing 
f{x)dx = pr{X 6 (x,x+dx)} 


If X is an rv, then so is MX) for any real-valued function ij/. IfX is discrete, 
then so also is i//(X), and we define the expectation (or expected value ) of 
MX) by 

E il/(X) = £ Mx)prlX = x) (2.3) 

xeS 


Similarly, if X is continuous, the expectation of MX) is 


EMX)= f Mx)f(x)dx (2.4) 

The expectation of X, EX , is called the mean of the distribution of X, and 
is generally denoted by p. By taking i p{x) = ax+b, where a and b are con¬ 
stants and expanding the right sides of equations 2.3 and 2.4, we find that 


E(aX+b) = aEX+b (2.5) 

The variance of the distribution of X, VX, is defined as E(X-/t) 2 , where 
p = EX, and is generally denoted by o 2 . We can write \X in the con¬ 
venient form 


VX = EX 2 -(EX) 2 


( 2 . 6 ) 
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Problem 2.1 If rv X is <?(A), find EX, VX. 

Solution. From equations 2.2 and 2.4, 

EX = j* xXe~ 2x dx = 1/A 
EX 2 = J* x 2 Xe~ 2x dx = 2/A 2 

Therefore 



Problem 2.2 Find EX and VX if rv X has the Bernoulli distribution with 
parameter 0. 

Solution. We write: X is &(0). The parameter is called the probability of 
success, since the Bernoulli distribution is often used as a model for experi¬ 
ments which have only two outcomes: success and failure. If the outcome 
is success we set X = 1 , if failure then X = 0. Other applications are 
when the outcomes are on and off, yes and no, or heads and tails. The 
probability distribution of X is therefore 

prl* = 1 ) = 0, pr(X = 0) = 1 -e 
From equations 2.3 and 2.6 

EX = 1 xpr(X = l)+0x pr(X = 0) = pr(X = 1) = 0 (2.7) 

EX 2 = 1 2 x pr(X = l)+0 2 x pr(X = 0) = 0 

Therefore 

vx = ex 2 -(ex ) 2 = e-e 2 = 0(1-0) □ 

Problem 2.3 Construct a model for the experiment in which a fair 
coin is tossed n times. 


Solution. Consider the experiment, or trial, in which a coin is tossed just 
once. Then the large experiment is a sequence of n of these trials. The out¬ 
come of a trial can be either *a head shows’ or ‘a tail shows’. We shall label 
these H and T respectively, as a convenient abbreviation. Then, since the 
coin is fair, pr(//) = pr(T) = Let us define an rv X which takes value 1 if 
H, and value 0 if T Then X is i#(j). The trial is called a Bernoulli trial. 

Now suppose that we repeat this trial n times under identical conditions; 
that is, we carry out a sequence of n independent Bernoulli trials. (Inde¬ 
pendence will be defined formally later.) We then define rvs 


X 


k 


1 if H on fcth trial 
0 if T on fcth trial 


(k = l,2,...,n) 


where each X* is independently mi). The sequence of rvs X„ X 2 . X n 
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describes the coin-tossing experiment. □ 

If we are interested in the number of heads that will be observed in the 
experiment, we define rv 

Z, = Xj + .-. + X, 

which, as the sequence of trials proceeds, increases by one every time the 
coin comes down head uppermost. We note that as n increases, sequence 
{Z n } is a stochastic process. It is a very important process, being the 
basis for random walk models. 


2.2 Bivariate distributions The joint distribution function of a pair 
(X, Y) of rvs is the function F XY of two real variables defined by 

F(x,y) = F XY (x,y) = pr(X x, Y < y) 

It is the probability that the value of rv X does not exceed x and the value 
of rv y does not exceed y. The subscripts will be used when we wish to 
avoid ambiguity; for example, in Section 2.1 for the rv X we could have 
used f x , F x , p x , <r 2 , E x , V x . For (X, Y), 

F(x, oo) = pr(X < x, 7 < oo) = pr(X ^ x) = F x (x) (2.8) 

is called the marginal distribution of X. If 

F(x,oo)F(oo,y) = F(x,y) (2.9) 

for every (x, y), then rvs X and Y are said to be independent. 

If X and y are discrete rvs which take values in sets S x and S Y respect¬ 
ively, then, from equation 2.8, the marginal distribution of X is 

pr(X = x) = X pr(X = x, Y = y) (x e S x ) (2.10) 

yeSy 


The conditional probability that X takes value x, given that Y has taken 
value y e S Y , is defined by 

pr(X = x, y = y) 


pr(X = x| y = y) = 


pr(y = y) 


From equation 2.10, therefore. 

pr(X = x) = £ pr(X = x| y = y)pr(y = y) 

y*Sy 


( 2 . 11 ) 

( 2 . 12 ) 


A conditional distribution has all the properties of a probability distribu¬ 
tion, and so we can define the expectation of X given that Y = y. 


E m = y X = X xpr(X = x\Y = y) 

xeS x 

If X and y are independent, then, from equation 2.9, 
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pr(AT = x, Y = y) = pitY = x)pr(y = y) (for every xeS x , ye S Y ) 
and so, from equation 2.11, 

pr(AT = x | Y = y) = prfX = x) 

A joint df F xr has a density if there is a nonnegative function f x y of 
two real variables such that for every (x, y) 

F x.r( x >y) = f* P f XY (u,v)dvdu 

J — oo J — oo * 

Then 

f X Y (x, y) dx dy = pr{X £ (x. x + dx), Y e (y, y + dy)} 

The marginal density of X is 

/*(*) = f°° fx.r^y)dy 

J ~ 00 

The conditional density of X given Y = y is 

fx |r-,(*) = (if/ y (y) > 0) (2.13) 

The value y is given and so is a known constant. The expectation of X 
given Y = y is 

^X\Y = yX = | -oo X /x|r*y( X ) (2.14) 

If X and Y are independent then, from equation 2.9, 

/x.y( x> y ) = /jM/yOO for every (x, y) (2.15) 

so, from equation 2.13, 

fx\Y= y ( x ) = /*( x ) (2.16) 

Now 

E x r iP(X, Y) = (“ J” ^x,y)f XY (x,y)dxdy (2.17) 

00 ^ * 00 

or 

Z Z ^(x,y)pr(X = X, y = y) 

xeSx yeSy 


We define the covariance of rvs X and Y to be 
cov(A', Y) = E x y {(X - E x y AT)(y - E x r y)} 

= E xy (*y)-(E x A-)(E y y) (2.18) 

Problem 2.4 Bivariate rv ( X , Y) takes values (-2,4), (-1,1), (1,1), (2,4), 
each with probability * (i) Find the marginal distributions of X and Y, 
and calculate covfX, y). (ii) Show that X and Y are not independent. 
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r Solution, (i) The marginal distributions of X and Y are: 

pr<* = -2) = MX = -2, y = 4) = i 
pit* = -1) = pr(X = 1) = pr(X = 2) = i 

similarly; 

pr(y = 1) = pr{(X = -1, y = l)or(A- = 1, Y = 1)} = i+J = \ 
P r(y = 4) = i 
similarly, 

EX = i{(-2)+(-l)+l+2} = 0, Ey = *(1+4) = § 

and 

E(Xy) = i{(-2x4)+(-l xl)+(lxl)+(2x4)} = 0. 

Therefore 

cov(x, y) = E(xy)-(EA-)(Ey) = o 

(ii) 0 = pr(X = -2, Y - 1) # pr(AT = -2)pr(Y - 1) = - J 

Therefore, X and Y are not independent. We note that Y = X 2 . □ 

If X and Y are both continuous rvs, from equations 2.17 and 2.13, 

E x y \//(X, Y) = [_ x {_ ^ 'l'(x,y)f X \ Y=y (x)f Y (y) dxdy 

= 'Kx,y)f X \, m ,(x)dx}f 1 {y)dy = E y {E x|y=y ^(2f, y)} 

Similarly, if one or other or both of X and Y is discrete, 

E x y iHX, Y) = E Y E x „ mY iHX, Y) (219) 

This will be called the decomposition rule and is the most important single 
device we shall use in our treatment of stochastic processes. For example, 
if (Z B } is the path of a particle, then to determine Ei l/(Z n ) we can condition 
on the position Z k (k < n) at an earlier time, i.e. 

mz„) = E ZkZ >(Z )= E Zk E Zn|Zit=Zk ^(Z ) (2.20) 

A special case of equation 2.19 is when i j/ depends only on X. Then 

E XY ilr(X) = E x E y(x=Jt . MX) = E x m (2.21) 

since, given X = X. ijj(X) is a constant, so 

E r , x .,p = m 

This result was used in equation 2.18. 

Problem 2.5 If the distribution of X | Y = y is &(0(y)), show that the 
marginal distribution of X is &(E r 6(Y)). 

Solution. From equation 2.7, 

E x|r=,* = P r <* = l|y = y) = 0(y) 
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Therefore 


pr(X = 1) = E x X = E y E x|y=y X = E y 0O') 

Clearly X can take only 2 values, 0 and 1; therefore 

pr(X = 0) = 1 —pr{X = 1) = 1-E y 0(y) 
so rv X is #(E y 0(y)). □ 

Problem 2.6 Show that 

V x r <HX, Y) = E y V z|r _ y *(*, Y) + V y E xlTmr +iX, Y) (2.22) 

Solution. We shall abbreviate {//(X, Y) to 4*- By equation 2.6, 

v XY \f/ = e xy 4 / 2 -<e xy 4 /) 2 

= {E y E xiy=y ^ 2 -E y (E x|y< . y tf0 2 } 

■b {E y (Ejj.| y=y 4*)~ ~ (E y E y | y=y 4*) } 

= E y V X | y _ y i/^ + V y E X | y=y i^ 

where we used equation 2.19 and introduced two middle terms which 
cancel. □ 

Problem 2.7 Show that if X and y are independent rvs, then for any 
real-valued functions 4> and i// 

e xy {<mx)4/(Y)} = {E I W){E Y 4,(Y)} 

Solution. By equations 2.14 and 2.16 

E*l r.,<HW = E x m 

also 

E xr {<t>(X)4/{Y)} = e y e x{y=y {4>(X)iHY)} = e y iHY)e xiy=y 4,(X) 

by equation 2.5, since, given Y = Y, 4 / (Y) is a constant. Therefore 
E X ' Y \4>(X)4s(Y)} = E Y 4/(Y){E x 4>(X)} = {E x 4>(X)} {e y 4/(Y)} 
since, with respect to the marginal distribution of Y, E y <p(X) is a constant. 

□ 

2.3 Multivariate distributions Rvs X,__, X n are mutually independent 

if, for every subset, X t) ,..., X lk , of two or more, 

pr(X (i < x,, .... X ik x k ) = pr(X (i < x, )...pr(X |k < x k ) 

for every (x,,...,**) 

We can extend the definition of conditional probability to show, for 
example, that if y,,..., Y n are discrete rvs then 

pr(y 1 = y 1 , .... Y„ = y a ) 
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= pr(y, = >>, )pr(y 2 = y 2 1 y, = y, )pr(y 3 = y 3 1 y, = y,, y 2 = y 2 ) 
-pr(^ = yj^: .... Y n-1 = y»-i) (2-23) 

From this we can generalise the decomposition rule to 

Ey,.»•••» = ^yi ^y 2 |yi = yi ^y3iyi = yi.y2=y2 

•• E y«|yi=yi.y«-»«yn-i »•••» ^»)* 

The result of Problem 2.7 generalises. 

Problem 2.8 Show that if X x , ..., X n are independently distributed 
rvs, then 

= Ex,.*.{*,<*, )...*.<*.)} = {E Xt <M*,)}-MExA(*,)} 

= A 1 A 2 ...A n , say 

Solution. We use the method of induction. By Problem 2.7 

B 2 = A t A 2 

Suppose 

B n -1 = 

Then, since X x ,..., X n are independent, 

Ex,...^r B .,|x„«x^2f 1 .•••,X n _ 1 )4>(X n ) = E x ,.>(X 1 

= 4*(x)E Xi .x„.,^2f 1 ,...,2f ll _ 1 ) 

Therefore 

= E x , E x .x..,,x„=xJ^ 1 (^ 1 )-^J} 

= {E,>J(^)}E X( . J[n _ g {+ l iX l )...t a . l VC^ l )} = A n B n _ x 

= ^2 •" ^B-I ) 


The result is thus proved. □ 

The converse is not necessarily true. In Problem 2.4 we had rvs X and Y 
which were not independent, but for which 

E x y (XY) = (E x X)(E y y) 

In a stochastic process {2f B } the states of the system, X 2 , X 2 ,... will 
in general not be independent, and we shall deal with consequences of this. 
In particular we shall study the special case in which X n depends on X n _,, 
but not on X n _ 2 , X n _ 3 , — 

2.4 Probability generating functions If X is an rv having discrete distri¬ 
bution p k = pr(X = x k )(k = 0, 1, 2, ...), then the probability generating 
function (abbreviated to pgf) of the distribution of X is 

G(s) = E x s x = p k s Xk 
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Now 


<5(1) = 1, G'(l) = 


dG(s) 


ds 


= EX, G"(l) = E{A(A - 1)} 


3 = 1 


SO 


(2.24) 


EX = G'(l), VA = G"(1)+G'(1)-{G'(1)} 2 
Problem 2.9 Find the pgf for an rv X which is #(0). 

Solution. pr(A = 1) = 0, pr(A = 0) = 1—0. Therefore 

G(s) = (l-0)s° + 0s 1 = 1 — 0+0s □ 

Problem 2 JO Find the pgf for an rv X which has a Poisson distribution 
with parameter k. 

Solution. We write that X is &(k). Here 

;*e -A 

P* = pr(A = k) = (k = 0,1,2,...) 

where k\ = 1 x2x3x...x/t, and, by convention, 0! = 1. Then 

Gis) = I PS = Z = e- 2 1 ^ = e-V* = e'* 1 "* 

k=o * k=o kl k =o k\ 

(2.25) □ 

Problem 2.11 Find the pgf for an rv A - which has a binomial distribution 
with parameters n and 0. 


Solution. We write that X is Bin(n, 0). Here 

p k = pr(X = k) = Q^l - 0r k (k = 0,1,..., 


n) 


where 


(2.26) 


0- 


n! 


k'.(n-k)! 


So 


<*> - .1 **■ - .1 . ( i t (;)(«! - W-* 


= {0s+(l - 0)} n (by the binomial theorem) 
= (1 — 0+0s) n 


□ (2.27) 


Problem 2.12 Prove that if rvs A,, ..., X n are mutually independent, 
and if X k has pgf G k (s) (k = 1, 2, ..., n), then rv Z n = Aj+.-. + A,, has 
Pgf 
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Solution. 


w = ri G k ( S ). 

k = 1 


i/ B (s) = E z y- = E X1 . jr y.+-«- = E*, = n G k (s) 

by Problem 2.8. □ 

As a consequence, if X k is (P(k k ), then Z is &\ V /. ) (from equation 

" V‘-> / 

2.25); and if A k is Bin(m t , 0) then Z n is Bin ( £ m k , Oj (from equation 2.27). 

For the coin-tossing Problem 2.3 we may treat Z n as though it were 
A, +... + A n , where the As are independently JS(0), i.e. independently 
Bin(l,0). 


Problem 2.13 If the conditional distribution of rv A, given that N = n, 
is Bin(n, 0), and if N is Bin(w, a), what is the marginal distribution of A? 

Solution. By equation 2.27, 

Gx\n=J- s ) ~ n=h sX = (1 — 0+0sy 

and 

G^w) = E n w n = (1 — oi+aw)" 

Therefore 

G x (s) = E x s x = E X N s x = E a E X|N=w s x (by equation 2.21) 

= E N (l-0+9s) N = G„(l-0+0s) = {1 —a + a(l —0+0s)} m 

= (l-a0 + a0sr 

We recognise that this is the pgf of a Bin(m,a0) rv. □ 

Problem 2.14 If A,, A 2 , ... is a sequence of independent rvs, each 
having pgf G(s), find the pgf for Z N = A, + X 2 + ... + X N , where N is an 
rv having pgf H(s). 

Solution. By Problem 2.12 

^Zk|n=» sZ ' = ^z„ s 7 ' = {G(s)} n 

Therefore 

E Zn s*" = E n E Zn|w=w s z " = E„{G(s)}* = tf{G(s)} 

since 

n(s) = e„s*. □ 

Problem 2.13 is a special case of Problem 2.14 in which the X k are 
independently £8(0) rvs. 
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2.5 Characteristic functions The characteristic function (abbreviated 
to cf) for an rv X is 

<*>(0)=E x e ,#x (i = V(-l), — oc < 0 < oo) 

There is a one-one relationship between the df F and the cf <f> for a rv X. 
If X is a discrete rv having pgf G, then 

<t>(0) = E(e' 8 ) x = G(e i8 ) 


so the techniques developed in problems about pgfs can also be used with 
cfs. For example, as in Problem 2.12, if X,, X n are independent rvs 
having cfs <^(0), 4> n (0) respectively, then Z n = X l +... + X n has 

cf {<f> l (9)...4)Jl0j}. Also, as in Problem 2.14, if X,, X 2 ,... is a sequence 
of independent rvs each having cf <f>(9), then the cf for Z N = Xj + ...-1-X^, 
where N is a rv having pgf H{s) is 

Ee i8Z * = E N {<f>(0)} N = H{m) (2.28) 

The moments of the distribution of a rv X are given by 


m = 1 , cf>'( 0 ) = 


dm 


dO 


= iEX, tl>"(0)=-EX 2 


a = o 


so 


EX = — i 4>'(0), VX = _^"(0)+{^'(0)} 2 


Problem 2.15 Suppose that n = EX and a 2 = VX, and define rv 
Y = (X — n)/a. Then E F = 0 and VP = 1. Find the relationship between 
<f> x and (f> r . 

Solution. We can write X = /i + oY, so 

<t> x (9) =• E x e i8X = E y e i8( " +,,1 ' ) = e“»‘E r e Uter = <t> y (9a) □ 

This result enables us to determine 4> x from the often well-known standard 
form (j ) y . 


Problem 2.16 Find the cf for the rv X which has a normal distribution 
with parameters n and a 2 , i.e. having density 

<-»<*<*) < 229 » 

Solution. We write that X is N(/r, a 2 ). Then rv Y = (X - n)/o is N(0,1), 
and 

friy) “ ^ij exp ( - 2^) (-<» < P < oo) 


Then we can show that 
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0y(0) = e"* 1 


Therefore 


4> x (0) = e ie ^ y (0 ff ) = = exp(i 9n-\9 2 a 2 ) □ 


EXERCISES 

1. If rv X is the value shown by a fair die, find EX, VX and pgf G(s). 

2. Deduce formula 2.15 from formula 2.9. 

3. If X and Y are Bernoulli rvs, show that Z = XT is also Bernoulli. 
If X t , ..., X n are Bernoulli rvs, show that Z n = XjXj.-.X^ is also 
Bernoulli. 


4. Given 



f (y) = 5 2y/ * 2 (°<y<*) 

nx-x jo (otherwise) 


f {x) = I 4 * 3 (0 < x < 1) 

(0 (otherwise) 

show that 


(i) 

/ (xv) = | 8x3 ' (° < P < X < () 
x ' r ’ (0 (otherwise) 

(H) 

f (vl = I 4 ** 1 ~ y2) (° < V < 1) 

Y jo (otherwise) 

(iii) 

f M N 1 -/) (y < x < 1) 

(0 (otherwise) 

(iv) 

Note that in (iv). 

pr x y (X > 2Y) = i 


pr x y (X > 2P) = JJ f XY (x,y)dxdy 

(x.jr:x>2y) 


5. If rv X has distribution pr x (X = 1) = f, pr x (X = = *, what is 

the distribution of rv Z = 1/X? What is the joint distribution of X and Z? 
Show that cov(X,Z) = — |. 

6. Prove formula 2.19 when X and Y are both continuous, and when X 
is discrete and Y is continuous. 


7. If rv X has pgf G(s), verify that VX = G''(l)+G'(l)- {G'(l)} 2 . 

8. If rv X is 0>(A), show that EX = VX = A. 

9. If rv X is Bin(n,0), show that EX = n0, VX = n0(l — 0). 
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10. The rv Y which is the number of independent Bernoulli trials 
(probability of success 0) up to but excluding the first success has a 
geometric distribution with parameter 0. We write Y is 5?(0), and 

pr(T = y) = (1 — Oy'6 (y = 0,1,2,...) 

Show that y has pgf G(s) = 0/{l—(1 —0)s} and that EV = (1 — 9)19, 

vy = (1 -0)/0 2 . 

11. The rv W which is the number of failures in a sequence of independent 
Bernoulli trials (probability of success 9) prior to the nth success has a 
negative binomial distribution with parameters n and 9. We write W is 
NB(n, 9), and 

pr(W = w) = (” + ^ -i y(l-0r (w = 0,1,2,...) 

Suppose that Y x , Y 2 ,..., Y n are independent $(6) rvs. By writing 

W = Y 1 + Y 2 +...+ Y n 

deduce that W has pgf G(s) = [0/(1 -(1 — 0)s}]", and that 
E W = n(l — 0)/0, VW = n(l — 0)/0 2 


12. If rv X is ^(A), and if the conditional distribution of Y given that 
X = x is Bin(x, 0), show that the marginal distribution of Y is 2?{XQ). 
(Use Problem 2.14.) 


13. 

14. 

15. 


Show that V(aX + bY) = a 2 VX+b 2 VY + 2abco\(X, Y). 
If rv X is (?(A), show that its cf <f>(0) is A/(A- i0). 

An rv y having density 


fy(y) = 


1 ° 


A_ 

W 


-^-(Ay)‘ , - 1 e-' 1 '’ 


O'* 0) 
O' <0) 


where T(a) = }”(/""" ’e - " du, has the gamma (a:/) distribution. Show that 
y has cf {A/(A — i0)}®. An /(A) rv therefore has a gamma (1; A) distribution. 
Show that Ey = aA, VY = aA 2 . 


16. If X ,,..., X n are independent rvs, where X k is gamma(a k ;A), show 
that Z n = X , +... + 2f B is gamma ^ £ «*: a\ 
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Chapter 3 

The Random Walk 


3.1 The unrestricted random walk 

Problem 3.1 Consider the stochastic process which is the path of a 
particle which moves along an axis with steps of one unit at time intervals 
also of one unit. Suppose that the probability is p of any step being taken 
to the right, and is q = 1 - p of being to the left. Suppose also that each 
step is taken independently of every other step. Then this process is called 
the unrestricted random walk. If the particle is in position 0 at time 0, 
determine the probability that it will be in position k after n steps. 


Solution. Let {Z n } be the stochastic process, where Z n is the position of 
the particle at time n, that is, after n steps from its starting point, 0. This 
stochastic process has a discrete time parameter space {0,1,2,...) and a 
discrete state space { — oo,..., - 1,0, l,...,oo). Now each step X is an 
independent rv having distribution 

pr(X = 1) = p, pr(X = -1) = q 
Initially Z 0 = 0. After n steps 

Z m = X t +X 2 +...+X m (3.1) 

where each X i is independently distributed as X. We can write equation 
3.1 as 

Z.-Z-1+X. ( 3 - 2 ) 

where X n is independent of Z n _,. 

We wish to determine the value of 


Let rv 


= Pr(Z„ = k\Z o = 0) 

y.J 1 = 1 (i=12 

' lo if* ( =-i 0 


(3.3) 


n) 


That is, let rv y. = £(A\+ 1); then each y, is an independent Bernoulli 
trial with probability of success p. Then rv R n = y, + ...+ Y n = ~(Z n + n) 
is Bin(n,p). Therefore, by equation 2.26 

Pol = P^ z . = k\Z o = 0) = pr {R n = %Z n + n) = %k+n)} 

(%kln)) P * k *' )qiim - k ' (K*+"> eS = {0,l,2,....n}) 


(otherwise). 


Since ER = np, VR n = npq. 
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EZ n = E(2R n — n) = 2E R„ — n = n(p—q) (since p+q = 1) 

VZ„ = V(2 R„-n) = 4VR n = 4 npq □ (3.4) 

We can call upon some quite deep theorems of probability theory to 
obtain the behaviour of Z n when n is large. For example, by the strong 
law of large numbers, with probability one 
1 „ 1 

~ Z n ~*~ EZ n = P-4 aS M-X 

That is, for large n, the particle will, if p > q, almost certainly drift in a 
positive direction along the axis of motion, the mean step length being 
p — q. Also, by the central limit theorem , 

w - = 5 ^5^~ anN(0 ' l|rvas " -c0 

So, from tables of the N(0,1) distribution, for large n. 

pr( — 1 -96 < W n ^ 196) * 0-95 

i.e. pr {n(p - <?) -1 967 (4npq) < Z n < n(p-q)+\%J(4npq)} a 0-95. 
Problem 3.2 Find approximate 95% bounds for Z 10 000 if p = 0-6. 
Solution, q = 0-4 

Therefore, EZ = 10 000(0-6 — 0-4) = 2000 

VVZ = 7(4 x 10000x0-6x0-4) = 79600 = 4076 a 98 
Therefore, 1-967 y Z a 1-96x98 a 192 

Therefore pr(2000-192 < Z 10000 sg 2000 +192) a 0-95 
i e. pr(1808 < Z 10000 < 2192) a 0-95 

A superficial acquaintance with the theory of statistics is sufficient to tell 
us that with n = 10000 the approximation will be very good indeed. □ 

Problem 3.3 Find approximate 95 % bounds for Z 10 000 if p = 0-5. 

Solution. EZ = 0. 7VZ = 7(4 x 10000x0-5x0-5) = 100. Therefore, 
pr( — 200 < Z I0 000 < 200) a 0-95 □ 

Now 7VZ is maximum when p = f. Therefore after 10000 steps, which 
would allow the particle to be up to 10000 steps on either side of its 
starting point, it is, with 95% certainty, effectively restricted to within 
± 200 steps. 

Problem 3.4 Evaluate the generating function 

p o*(s) = i tip 

n = 0 
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for k = 0 and k = 1, where p™ is defined in equation 3.3. 


Solution. These are not pgfs, since pj> (n = 0,1,2,...) is not a probability 
distribution, it being possible for the particle to be in position k for many 
different values of n on the same random walk. For example, p^ = 1, 

Poo = (j) pq = 2pq ' s0 \ ^ 1 + 2 P4 > ! - 

^oo(s) = I (,”)p*V B s"= £ M(MS 2 r 

n = 0 , 2 . 4 ,... \2 n / m = in = 0 \ rn / 

= (l-4pqs 2 ) _i = m(s), say (3.5) 


(We might recognise this as the Legendre polynomial generating function.) 
Now 


2n + \\ = ( 2 n+ 1 )! = 2 (n + l)-l ( 2 w)! = /_ 1 _V 2 «\ 

v n ) (n+l)!n! n + 1 n!n! \ n + \J\n) 


(3.6) 


and 



Therefore 


-icr)r—rtiCH* 



— - - (by comparison with equation 3.5) 

7(1-4x) 

- 7(1-4w)} (3.7) 


p„w- f - i ( 2 "7) p " Vsi "‘ 

n = 0 " = °\ n / 

- i 3 - 6 > 

= 2ps(l-4pqs 2 )-*-^{l-(l-4p^ 2 )*} = 2^W S )-1} 


(by equations 3.5 and 3.7) □ 
We shall in future try to find solutions which are based on probability 
arguments rather than analytical techniques. 

In Section 4.4 we shall meet a formula by which, from P 00 (s) and P 01 (s) 
we can calculate P 0k (s) for any integer k. 


3.2 Types of stochastic process A process {Z,:teT S (— oo, x)} such 
that for any {r„t 2 ,...,t B } e T, where t, < t 2 < ... < £„, the rvs 

Z t 2 ~ Z h' Z l 3 ~ Z tl . Z t„~ Z t n - l 
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are independent, is called a process with independent increments. 

A stochastic process {Z,:teT = (- 00 , 00 )} is stationary if, for any 
{t, , t 2 , ..., t n } 6 T the joint probability distributions of 
Z 7 7 

and 

are the same for all r e( — oo, oo); and is stationary in the wide sense if 
the covariance function g(t. t + r) = co v(Z,,Z (+t ) = g(r), is a function of 
t only, for all t e T. 

A process with independent increments has g(t,t + t) = g(t) (t > 0). 

A Markov process is a stochastic process {Z, : t e T = (-oc, oo)} for 
which, given the value of Z t the distribution of Z s (s > t) in no way depends 
on a knowledge of Z u (u < t). The future behaviour, when the present state 
of the process is known, is unchanged by additional knowledge about its 
past behaviour. Thus if 


t 0 < t, < 


< T * < h < h 


< t_ 


then the joint distributions of 

Z t,’-"’ Z tJ Z ro = Z 0*”'» Z t k = Z k 

and of 

Z U ’"■’ Z l„l Z T k = Z k 

are the same. This is referred to as the Markov property. 

Markov processes are by far the most important class of stochastic 
processes and their theory is highly developed. Every stochastic process 
with independent increments is a Markov process. 

Problem 3.5 Show that the unrestricted random walk is a stochastic 
process with independent increments (and so is also Markov), and is 
stationary under translations by integer intervals. 

Solution. If {A*} is a sequence of independent rvs, then {Z n }, where 

Z n = £ X k ' has independent increments since if n 0 < n. < n, then 

k = 1 12 

*1 l »2 

Z n\ Z no = X X k and z n 2 ~ z n, = X X k are independent. 

k -i*o+ 1 k — ni + 1 

The position of the particle at time n is Z n and since the probabilities 
of stepping to the right or left in no way depend on Z n the joint distribution 
°f Z no , Z ni , .... Z„ k depends only on the time intervals between the 
observations, and so is the same as the joint distribution of Z no+m , 

z » 1+ m’---’ Z n lc+ « foral1 m -1,0,1,2,...). " 0+ n 
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3.3 The gambler’s ruin 

Problem 3.6 The random walk with two absorbing barriers. Two ad¬ 
versaries, A and B. have resources La and Lb respectively. They play a game 
in which each play results in A winning £1 from B with probability p, or 
B winning £1 from A with probability q = 1 -p. Each play is independent 
of every other play. Find the probability that A will eventually have to 
withdraw on losing his entire La. 


Solution. Define rv 



if A is eventually ruined 
otherwise 


Then rv Y has a Bernoulli distribution with, as its probability of success, 
the probability that A is eventually ruined. We consider the random-walk 
process {Z n : n = 0,1,2,...}, where Z n +a are the resources (in £) of A 
after the nth play. Then Z 0 = 0 and the state space 

S = { — a, — a + l,..., — 1,0, 1,2,...,b— 1,6} 

The game will cease when A has won B’s Lb, or when B has won A’s La, 
i.e. when Z n first reaches -a or b: thereafter Z n+1 , Z n+2 ,... will all take 
the same value as Z n . These states —a and b are called absorbing states. 

Rv y = 1 if Z n reaches —a before it reaches b, and Y = 0 otherwise. 
We seek pr(T = l|Z 0 = 0). 

Let us write 

G,(s) = E r|Zo .,s r = 1 -0,+e t s = 1 -(1 -s)0 { (i e S) 

the pgf for the Bernoulli rv Y given that the walk starts in state i, where 
0 ( = pr(y = l|z 0 = 0. 

We find a set of recurrence relations for the G ( , which we solve for 0 o . 
Clearly 8_ o =1, 0 b = 0. 

We need some preliminary results. Clearly 

P r ( T = 11 Z„ = i) = 0, 

irrespective of n. Also, since the plays are independent, the random walk 
has the Markov property, so 

^r|Zi-*.z 0 =i sr = Ey|Zi=* sy = G k (s ) ( i,keS) 

Then, by the decomposition rule, 

Gj(s) = E y | Zo= jS > ’ = Ejr,z,|Zo»i sr = ^Zi|z 0 =i^r|Zi=Zi.Zo=i sy 

= ^Zi|Zo-i^Z|^ 

Therefore 

I - @i = G,(0) = Ezi|z o *f^Z|(0) = E Zl | Zo=i (l — 0 Z( ) = 1 — E Zi | Zo=j 0 Zl 
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so 


“ ^Z l |Zo = i^Z, 

= 0,-ipr (Zj = i — 11Z 0 = i)+O i+1 pr(Z l = i+l|Z 0 = /) 

= <70i-i + P0f+i O' = -0+l,...,i>-l). (3.8) 

This is a sequence of a + b — 2 second-order difference equations in 
a + b - 2 unknowns. These equations can generally be written down directly 
with just a brief explanation. The use of generating functions introduces 
an extra flexibility in allowing us to abstract from our results any of those 
aspects of the process which interest us. In this problem there is a one-one 
correspondence between G. and 9 t , so this is unnecessary. The decomposi¬ 
tion rule applied in this way is sometimes referred to as a decomposition 
based on the first step. We now solve the equations for 0 Q . We write 
equation 3.8 in the form 

(p + q)6 i = q6 l _ l +pO i+l 

SO 

M+1 ~ d i ) = (* = -a + l,...,b-l) 

Write /, = 0, — d,_ v and A = q/p. Then 

I. = XI i _ 1 = A(AJ,_ 2 ) = ... = A'/ 0 

Then 


-i = e b -e_ a = £ Z /, = / 0 Z 

i= —a+1 i= -o+1 -a+ 1 

and similarly, by writing a = 0, 

-0 o = 0*-®o = 'ol* 

i»l 

SO 


00= Z A 1 / I A‘= 

1*1 1* -a+ 1 


(A = 1, i.e. p = q = 


a+b 

U = q/p * 1) 


□ (3.9) 


Problem 3.7 What is the probability that /l is eventually triumphant? 
Solution. Let rv 


and let 


Y* = 


1 if A is eventually triumphant 
0 otherwise 


<t> { = pr(y* = 1|Z 0 = o 

= pi {A is eventually triumphant | A starts with £(a + i)} 
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= pr{fi is eventually ruined | B starts with £(b —i')} 

Then we would obtain the same equations as in Problem 3.6, but with a 
and b interchanged, p and q interchanged, and i changed to — i. Thus, 
from equation 3.9, 


<£o 


r a 
a + b 
1 —A” 
.1 —X a+b 


(A = 1, i.e. p = q = £) 
(A = q/p # 1) 


□ 


Problem 3.8 Show that it is certain that the game eventually ends. 

Solution. The probability that the game eventually ends is the probability 
that A is either eventually ruined or triumphant 

= pr(y=l|Z o = 0) + pr(y* = l|Z o = 0) 

= 0Q + 0O = 1 ( a h P>4 > 0 f° r which P + 4 = 1) 

Thus with probability 1, the game will eventually end. That is, for the 
random walk with two absorbing barriers, absorption is certain with 
probability 1. □ 

Problem 3.9 Find the distribution of the duration, T, of the game. 
Solution. Let 

F,(s) = E r|Zo=J s T = t /<"V 

n = 0 

where 

f\ H) = pr(T = n | Z 0 = i) = pr{game ends on nth play | A starts 

with £(a + i)} 

Given Z x = j, then by the Markov property and time independence 

T = 1 + T 

where T' has pgf Fj(s) 

'• e - E t\z,=j.z 0 =i sT = ^r|z 0 =^ sI + r = sE /( s ) 

Then, by a decomposition based on the first step, 

F.(s) = E T Zi | Zo=l .s T = E Zi | Zo , J E r|Zi=Zi Zo=( s r 

= sE ziiz 0 =( ^Z\ ( s ^ = E,_ ,(s)+pF (+ ,(s)) (i = — a + l,...,b— 1) 

We have the boundary conditions that 

F-JL') = E r|z„=-o sT = Es° = 1 
and F b (s) = 1 similarly. We seek F 0 (s). The general solution is 
F.(s) = +(s) {a(s)}' + B(s){P(s)}‘ 
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where a(s) and p(s) are the roots {1 T,/(l — 4pqs 2 )}/2ps of the quadratic 
equation 

v = s(q + pv 2 ) (3.10) 

and /4(s) and B(s) are determined from the boundary conditions. Then 
F_ a (s) = 1 = AoL~ a + Bp- a 
F b (s) = 1 = A<x b + BfP 

have solution 

P b -P~ a a 5 -a 

A ~ a~ a p b -(x b p~ ay B ~ ~ct~ a p b — 0L b p~^ 

Then F 0 (s) = A + B. □ 

When s = 1, a(l) = 1 and /?(1) = A = q/p\ and for A # 1 we find 
F 0 (l) = 1. When A = 1 we can set P = a(l+e) in the solution and let 
e -+ 0. Since 

^oO) = i ft = i 

n = 0 

pr(T >k\Z o = 0)= £ ft^° as k 

n = k 

With probability 1, therefore, the game eventually ends. This is another 
solution to Problem 3.8. 


Problem 3.10 The random walk with a single absorbing barrier. What is 
the probability of /4’s eventual ruin if he is playing against a casino which 
has unlimited capital? 


Solution. If we let the casino be adversary £, and have unlimited re¬ 
sources, then the probability of ,4’s eventual ruin, given that he begins 
with £a, is 


lim 0 Q 


lim —— 

‘^£+1 

b 

II 

II 

lim l - X ' b 

(A = q/p > 

^00 \-X- a X~ b 

y /‘"I 

(A = q/p < 

1 (p *£ q) 


a 

V 

& 
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j since A -h —» 0 if A > 1, and A* -*■ 0 if A < 1. Hence, if p > q, there is a 

probability (q/pf that gambler A is eventually ruined, but there is prob¬ 
ability 1 — (q/pY that the game never ends. We say that the random 
walk has positive drift if p > q. On the other hand, if p ^ q (that is, if 
there is zero or negative drift ) gambler A will almost certainly be ruined, 
i.e. with probability 1 he will reach the barrier at — a. □ 

Thus if p ^ q, as it always will be in any casino, the gambler knows that 
eventually he will be ruined. However, what will be of interest to him is 
the distribution of the time before he loses all his money. We investigate 
this time to absorption in the next problem. 

Problem 3.11 The time to absorption. Find the probability distribution 
of the rv T a0 , the number of plays made before a gambler who starts with 
£a is ruined, given that p < q. 

Solution. For convenience we now take our single absorbing barrier to 
be state 0 (not state —a as before), and consider the stochastic process 
{Z n : n = 0,1,2,...}, where Z n is the capital of the gambler after the nth 
play. Then Z 0 = a. Now 

= E x|Zo=# s r = Es r - = lim F 0 (s) 

b-*CO 

where F 0 (s) is the solution of Problem 3.9. Suppose p < q, then, from 
equation 3.10, a(s)/J(s) = q/p > 1. Therefore |/?(s)| > 1, so {/?(s)} 6 is 
unbounded as b -* oo. Therefore, since F b (s) is bounded as b —► oo, 
B(s) = 0. Then 

F 00 (s) = lim F_ a (s) = 1 = -4(s){a(s)} -<1 

b~* oo 

SO 

FJs) = lim F 0 (s) = ,4(s) = {a(s)}» = j 1 ~ V ( 1 - 4 P^) | a 

b~* oo ( 2pS J 

This also holds when p = q = □ 

Alternatively we note that for £a to be reduced to £0 for the first time, 
it must be reduced successively for the first time to £(a-l), to £(a-2), 
and so on to £1 and finally to £0. During each of these unit reductions the 
gambler’s capital can rise, but eventually, since p < q, it will fall. Let 
rv T.j (j < i) be the number of plays for the gambler’s capital to be reduced 
for the first time from £i to £j. Then 

T tt0 = T at ._ 1 + T a _ lra _ 2 +...+ T l0 (3.11) 

But the time taken to lose a pound does not depend on which pound it is, 
so by this time independence each is distributed as T 10 , and these 

are mutually independent by the Markov property. Thus 
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FJL*) = {FJs)}° 

We need therefore determine only F, 0 (s). We find an alternative derivation 
of this in Problem 4.18. 

Problem 3.12 The random walk with a single reflecting barrier. Consider 
a random walk such that the particle, when it reaches state 0, moves on the 
next step to state 1 with probability 1, and resumes its random walk 
behaviour (Figure 3.1). Then state 0 is called a reflecting barrier. 



Figure 3.1 The random walk with a single reflecting barrier 

If this random walk process {Z B : n = 0,1,2,...} starts in state i 
(i e S = {0,1,2,...}), determine a set of difference equations for the pgf 
/7}"’(s) for the position of the particle at time n. 


Solution. We consider a decomposition based on the last step. 
n\ n+l \s) = E 2iiti | Zo= ,s z " tl = E Zn|Zo=j {E Znti|Z(i=ZnZo=i s z "*'} 

Now 

Z n + 1 with probability p (Z B # 0) 

Z B+1 = • Z n — 1 with probability q (Z B # 0) 

.1 (Z„ = 0) 

so, by the Markov property, 

P pZn+l — p c^w+l 

C Z„♦ I |Z„ = Z„.Z 0 = 1 ” ^Z„ ♦ I |Z n = z n * 




ps 7 "* 1 +qs z " 1 = ( ps+qs ^s 7 " (Z n # 0) 
s = (ps^-g^s“ 1 )s z ’ , ^-q(s-s" , ) ( Z n = 0) 


Therefore 


n\ n+1 \s) = q(s—s~ 1 )pr(Z (l = 0|Z o = i)+E Zn|Zo=i (ps+qs _, )s z " 

= qis-s-'WyM + lps + qs-W'Is) (3.12) 

The initial condition is /7} 0, (s) = E Zo)Zo=i s z ° = s'. □ 


28 


Problem 3.13 On the random walk of Problem 3.12, if p s* q, the particle 
is certain to return to state 0 from any positive state i (by Problem 3.10). 
It will therefore spend periods away from the reflecting barrier broken up 
by single steps onto it. By the Markov property, after first stepping onto 
the barrier, the process is in a statistical equilibrium, independent of the 
initial starting state, and the periods off the barrier are independent and 
identically distributed random variables. Find the equilibrium distribu¬ 
tion of Z B as n -*• oo, and show that this solution exists only if p is strictly 
less than q. 00 

Solution. Let /7‘"’(s) - /7(s) = £ 7t.s J as n -* oo. Then, letting n -» oo 
in equation 3.12, j=0 


77(s) = (s-s -1 )q77(0)+ (ps + qs l )Il(s) 


i.e. 


where v = plq 

1-(ps+qs *) 1—vs 

Now /7(11 = 1. Therefore. 1 = 2jt„/(l—v), so 77(0) = n n = 1(1 —v) = 


1 — 1 /( 24 )- 

We note that if v = p/q = 1 then n 0 = 0; and so 77(s) = 0, which is 
not a pgf. Similarly if v > 1, i.e. p > q, then n 0 < 0, so n 0 is not even a 
probability. If v < 1 then 


n(s) = 


(l-v)(l+s) 
2(1-vs) 




— n o n v (s)+(\ n 0 )n w (s) 

where n^s) = 1 is the pgf of an rv U which is 0 with probability 1, and 
Fl w (s) = (1 - v)s/(l - vs) = s Tlifs) is the pgf of an rv W = V + 1. where V 
has a geometric distribution with parameter 1-v = 2-q" 1 . The rv U 
corresponds to the time on the barrier and W to the time off it. □ 

The form of 77(s) is that of a mixture of the pgfs of the two rvs U and W. 
A mixture is the result of a two-stage experiment. First we decide according 
to the result of a Bernoulli trial with probability of success n 0 whether to 
observe rv U or rv W. If a success results then we observe U, otherwise W. 
This model has the same probability behaviour as the equilibrium 
distribution of the random walk. 

We shall be meeting a theory for equilibrium distributions later. 


Problem 3.14 Consider a random walk on (0,1,2,..., a), where at each 
step p = q = y State 0 is absorbing, and state a is impenetrable with 
pr(Z. + 1 = a|Z„ = a) = i pr(Z B+1 = a-1 |Z B = a) = \. (That is, if 
the particle is in an impenetrable state, then at the next step with some fixed 
probability 0 it remains in that state for a further time unit, otherwise with 
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probability 1 — 6 it retreats a step and resumes its random-walk behaviour.) 
Show that if the particle starts in state i(^0) then absorption is certain, 
and determine the expected time to absorption. 

Solution. We consider a decomposition based on the first step. Let F k (s) 
be the pgf for the first passage time T k from state k to state 0. Then, for 
k = 1,2 ,...,a— 1, 

F k (s) = Es r - = Es 1 + T| ‘~ , pr(Z l = k-l\Z 0 = k) 

+ Es 1 + r ^*pr(Z, = fc +1 |Z 0 = k) 

= \sF k _ 1 (s)+\sF k+1 (s) (3.13) 

The boundary conditions are 

F 0 (s) = Es r ° = Es° = 1 (3.14) 

and 

F a (s) = Es 1+Ta ' , pr(Z 1 = a — 11Z 0 = a) 

-t-Es^^pitZj = a\Z 0 = a) 

= IsF'.^+LsF'M (3.15) 

We consider the case s = 1, then solving equation 3.13 successively 
starting with the last equation (3.15) we find F k (l) = 1 (k = 0,1,2,...,a), 
so absorption must take place with some finite n (with probability 1). 
(See Problem 3.9.) 

Let 

= E T k = j{F k (s)} 

*** S= 1 

then M k < oo and, from equations 3.13, 3.14 and 3.15, satisfies 
M k = 1+iM^+jA#^ (k = l,2,...,a — 1) 

Af 0 = 0 (3.16) 

We note that these are solved by M 0 = 0, M k = oo(k = 1,2,...,a), but 
since absorption occurs with probability 1 within a finite time, we can 
rule out this solution. Let I k = M k — M k - l (k = 1, 2,..., a), then equations 
3.16 reduce to 

/ * = / k + i +2 (* = l,2,...,a-l), I a = 2 

so 

/* = 2(a+l-fc) (fc= l,2,...,a) 

Therefore 
M, = Af,-M 0 = 

= • Z ** = 2 Z (a+l-k) = i(2a+l-i). □ 

*=1 *=1 
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Problem 3.15 A particle executes a random walk on the vertices of a 
cube. The steps are independent, and the particle always steps with 
equal probability to one of its three neighbouring vertices. If A and H are 
opposite vertices, find the pgf for the first passage time from A to //, i.e. 
the time at which a particle which sets out from A first reaches H. 

If A and H are absorbing states, what is the probability of absorption 
at H given that the walk starts at a vertex which is neither A nor //? 


Solution. Label the vertices of the cube as in Figure 3.2. 


0(A) _ 0(2) 




0(2) 


C( 1 ) 




O(U) ) 


Figure 3.2 

Define states for the system by the minimum number of steps from the 
vertices to A. Then if the particle is at A, it is in state 0; if at B, C or D it is 
in state 1; if at £, F or G it is in state 2; and if at H it is in state 3. Let 
be the first passage time from state i to state 3 (i = 0,1,2,3), then T 3 = 0. 
If F.(s) = Es r ', we seek F 0 (s). We consider decompositions based on the 
first step X. 

F 0 (s) = Es To = Es 1 + Tl = sF x (s) 
since from state 0 the first step must be to state 1. 

F,(s) = E x E TiiXmX s r ' = jEs 1+To +|Es 1+T2 = *sF 0 (s)+§ S F 2 (s) 

since, with probability the first step is to state 0 or, with probability 
to state 2. Similarly 

F 2 (s) = §sF,(s)+L;F 3 (s) = fsFjfsJ+L; 
since F 3 (s) = Es° = 1. We can solve these to obtain 

F 0 (s) = 2s 3 /(9-7s 2 ) (so ET 0 = 10) 

Now suppose that states 0 and 3 are absorbing. Let 0,(i = 0, 1, 2, 3) be 
the probability of absorption in state 3 if the particle starts in state i. We 
seek the values of and 0 2 . Clearly 0 o = 0, 0 3 = 1. By decompositions 
based on the first step, starting in state 1 and state 2 respectively, we obtain 
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“ 3^0 + 3 ^ 2 ’ ^2 “ 3^1 “^ 3^3 

SO = 5 » ®2 = 5 ^ 

3.4 Generalisations of the random-walk model Instead of just a single 
step at each time instant we can allow larger jumps. 

Problem 3,16 Suppose that in a random walk the steps are independently 
distributed as a discrete rv X, where pr(X = k) = p k (k = 
-1,0,1, 2, ...); and that there are absorbing barriers at integers - a 
and b(a,b> 0), the process being absorbed as soon as the walk reaches 
or passes over a barrier. Suppose that the process starts in state i . Find 
difference equations for 9 r the probability that absorption eventually 
occurs at —a. Find difference equations for F f (s), the pgf for the time to 
absorption of the process. 

Solution. We argue as in Problem 3.6. Define Bernoulli rv 

{ 1 if absorption eventually occurs at — a 
0 otherwise 

and consider the random walk process {Z n :n = 0,1,2,...}, where Z n 
is the position of the particle after n steps. We seek difference equations for 
0. = pr(7 = 1 1 Z 0 = i). We use a decomposition based on the first step: 

P r (Y = 1|Z 0 = 0^ 

= £ pr(Z, =j\Z 0 = i)pr( Y = 1 1 Z 0 = «\ Z, = )\ 

j- -00 

i.e. 

0 i = f. Pj-t e j (ie I = {-a+l,-a + 2,...,b-l}) 

j* “00 

by the Markov property. Clearly 

0-oo = ••• = 0-fl-i = 0-o = 0b = 0&+i = ••• = 0 qo = 0* 

Therefore 

- a 6—1 ao 

0 i = lx Z Pj-i+ Z p,-A+ ° x Z Pj-i 

j= -00 ' j*-o+l j=b 

= 9i+Y.Pj-t e j (<ei) 

where ' rl 

0/ = Z P* = pr(* < — a—i) 

k = j- i= — oo 

As in Problem 3.9, if rv T is the time to absorption, then if Z 0 = i 
(i e /) and Z, = j we can write T = 1 + T\ where, if j e /, f has pgf f\(s), 
but if j $ I then T' = 0; i.e. 
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E s r -E , + r _ \ sF j( s ) O' 6 /) 

c r|Zi«j.Zo»i 5 “ “ | s 

Therefore 

Ffc) = E Zl j Zo=!i E T | Z|=Zi Zo=i s r 

= 4 z” p^i + z pj-i^^)+ z Pj~\ oe/) 

U* “°° j«-fl+l j-b ) 

= s | 0 -* + Z Oe/) 


where 


jf = pr(X < — a — i or X ^ b — i) 


Problem 3.17 Consider the random walk in which the steps are in¬ 
dependently distributed like a continuous rv X, where X has density 
p(x) ( — oo < x < oo ). Find an integral equation for F z (s) = E nZo =I s r . 
where rv T is the number of steps before absorption, if the process starts 
at z (—a < z < b). 


Solution. Exactly as in Problem 3.16, 


where 


r z o)=4^ z ) + r_ 


p(y-z)F(s)dy 


’j (—a < z < b) 


g(z) = pr(X ^ —a — z or X ^ b — z) 


EXERCISES 

1. A particle at each unit time point makes a step to the right with 
probability p, a step to the left with probability q , or no step at all with 
probability r = l—p — q. Find the pgf, G n (s), for the position, Z n , of the 
particle after time n if initially it is in position 0. Find the mean and 
variance of this distribution. Show that 

G*(s,f)= £ G n (s)r" = s/{-ps 2 t+s{\-tr)-tq} 

11 = 0 

2. Consider a particle which takes an unrestricted random walk as in 
Problem 3.1. Suppose that the walk has positive drift (A = q/p < 1) and 
that the particle starts in state 0. Use Problem 3.10 to find the distribution 
of rv W, where — W is the leftmost point the particle reaches. 

3. Consider the special case of Problem 3.11 in which a = 1 and 
p = q = i. Expand F 10 (s) in powers of s to obtain a formula for 

p k = pr( T { 0 = k). Evaluate p k for k = 1,2. . 5. Describe a coin-tossing 

experiment by which this process can be realised, and carry out 10 
realisations. 
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4. A programmed learning text offers three alternative answers to each 
question. One of the answers is correct and scores two marks. The student 
selecting this answer can proceed immediately to the next question. 
Another of the answers carries one mark. The student selecting it can 
proceed to the next question only after a unit of further reading. The third 
answer carries no marks, and the student selecting it must undertake two 
units of further reading before proceeding to the next question. If the 
student makes a random and independent choice of answer to each 
questioa what is the pgf for the total number of units of further reading 
he must do before he selects an answer which carries two marks? 

5. Solve Problem 3.14 for the general impenetrable barrier having 
parameter 0. 

6. Solve Problem 3.14 for the case in which p — l —6, q = 0 and the 
impenetrable barrier also has parameter 0. 


n {l) = 
Pij 


Chapter 4 
Markov Chains 

4.1 Definitions A Markov chain (abbreviation: MC) is a Markov 
process {Z n : n e T) with a discrete time parameter space T. and a finite 
or countably infinite state space S. We take, without losing generality, both 
S and T to be subsets of integers. The time-independent transition prob¬ 
abilities of a MC are 

fit = P r < Z « + . =J \ Z n, = ') (U £ 

(independent of m). For example, for the unrestricted random walk. 

P O'= i+l) 

1 -p 0 = <-l) 

0 (otherwise) 

By convention we write p (j for p\V. The one-step transition matrix (or just 
transition matrix ) is the matrix 

P = (Ptj) iiJeS) 

Now p.j ^ 0, ]T p (j = 1; the first because the are probabilities, the 

}*S 

second because, given Z 0 = i, Z Y must be in some state j of S. For the 
same reasons the elements of the n-step transition matrix 

P= (P5P (UeS) 

satisfy 

o. 1/S = 1 

We call matrices with these properties—that is, square with nonnegative 
elements and unit row sums— stochastic matrices. 

Problem 4.1 What is the transition matrix of the MC {Z n } where each 
Z n is independently distributed as rv Z, which has distribution 
pr(Z = k) = p k (k = 0.1,2,...)? 

Solution. 

P tJ = pr (Z l = j\Z 0 = i) = pr(Zj = j) (by independence) 

= Pj 

Therefore 


...\ 


j 

= 0 

7 = 1 

7 = 2 

i = 0 1 

f Po 

Pi 

P 2 

i=l 

Po 

Pi 

Pi 

i = 2 1 

Po 

Pi 

Pi 


□ 
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Problem 4.2 What is the transition matrix of the MC {Z H } where 
Z„ = Z K _ 1 +X n (see equation 3.2), and the steps X n are independently 
distributed as rv X, which has distribution pr(X = k) = p k {k = 0,1,2,...)? 
What if X has distribution pr(X = k) = q k (k = ..., —1, 0, 1, 2,...)? 
Solution. 

Pij = pr(Z, = j\Z Q = i) = pr (Z,-Z 0 =j-i\Z 0 = i) 

= pr(X, = j—i) (since Z 0 and X x are independent) 

O' = i.i+M+2,...) 

0 = 0,1,2, ..., i — 1) 

or q j _ l 0 = ..., — 1,0, 1,2,...) 

so 


_ } P J-< 

\o 


P = 


7 = 0 

j = 1 

7 = 2 

7 = 3 ... 

i = 0 

/Po 

Pi 

Pi 

p 3 ...\ 

i = l I 

' 0 

Po 

Pi 

Pi - \ 

( = 2 

0 

0 

Po 

Pi • 

i = 3 \ 

V. 

0 

0 

, - 


or 



... ;= -2 ;= -1 

7 = 0 

7 = 1 

= _1 ) 

<■ • • 9-i 4o 

4i 

4 2 

= 0 

••• 4-2 4-1 

4 0 

4i 

= 1 

«N 

1 

in 

1 

4-i 

4o 

= 2 \ 

l" 4-4 4-3 

4-2 

9-i 


Problem 43 A rat is put in the maze illustrated in Figure 4.1. At each 
time instant it changes room, choosing its exit at random. What is the 
transition matrix of the MC {Z n }. where Z„ is the room the rat is occupy¬ 
ing during (n, n + 1)? 

Solution. 
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Figure 4.1 


Problem 4.4 Show that the factory described in Problem 1.7 can be 
analysed as an MC. 

Solution. Exercise 1 of Chapter 1 asked for a list of the possible transi¬ 
tions between the states of the factory. We list the transitions here, with 
their corresponding probabilities. They were found by considering the 
event at the factory which caused its state to change. Let q = 1 — p. 


Transition from state i: 

0 

0 1 

1 

2 

2 

3 

otherwise 

to state j : 

0 

2 0 

2 

1 

3 

2 


with probability p tj : 

4 

P 4 

P 

9 

P 

1 

0 




0 

1 

2 

3 


i.e. transition matrix 

0 

1 4 

0 

P 

°\ 


p = (Pu 

) = 

1 

4 

0 

P 

0 




2 

0 

9 

0 

P 




3 

\o 

0 

1 

0/ 



□ 


Problem 4.5 Consider the library book loan process described in 
Problem 1.5. Suppose that the times in weeks it takes the reader to read 
library books are independent rvs having distribution function F(t). If 
{F(j): j = 0,1.2,...} is a sequence with F(j) < F(j+ 1) show that the 
process is an MC, and find its transition matrix. 

Solution. Let the time to read a book be an rv T. A book is exchanged if 
and only if the reader finished it during the previous week. This has the 
Markov property, and so the process is an MC. The only possible transitions 
are i to 0 or to i+1 (i = 0, 1, 2, ...), so p tj = 0 0 # 0, i+1), and 
Pio +P M +1 = 1- Now P r (Z„ = 0 = pr(T > 0 = 1 — F(i), and 

P u+ i = P^+i = i+1 \ z n = 0 = pr(T > i+11 T > i) 

_ pr(T > i + l,T > ») _ pr(T > i+1) _ 1-F(i+1) 
pr(T > i) “ pr(T > i) 1-F(0 

(i = 0,1,2,...) 
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where F(0) = 0: and p il+ , < 1 since {F(j)} is a monotone increasing 
sequence. Therefore 


Pi o — 1 Pi.l+l 


F(i+\)-F(i) 

1-F(0 


These probabilities are the elements of the transition matrix. □ 


Problem 4.6 Consider a time-dependent MC {X B : n = 0,1,2,...} 
defined on the state space {0,1}. If pr(X 0 = 1) = 9 0 and 
pr(A: m = 11A" m _j = 1) = 0 m (m = l,2,...),showthat{Z n :n = 0,1,2,...}, 
where Z B = X 0 X l ...X H , is also a time-dependent MC, and determine 
the marginal distribution of Z n . 


Solution. 

Z, = Z'-iX„> z 0 = x 0 

If Z B _, = 0, then Z B = 0 (so state 0 is an absorbing barrier of the time- 
dependent random walk {Z B } on {0,1}). If Z B _, = 1, then X m = 1 
(m = 0, 1.n— 1), and by the Markov property of {X n } 

P'(Z„ = 1 \z,-i = 1) = P^„ = i 1^.-1 = D = o n 

and 

P^ b = O|Z b _ 1 = D=1-0 b 

Therefore, given Z n _ 1# we need no further information in predicting 
Z n , so {Z n } is an MC. 

Z n is a Bernoulli rv (Exercise 3 of Chapter 2). Also 
E 7 — 

n 'Z„\Z„- l =Z n - l Z ' n - 

= M.-I 

Therefore, the probability of success, 


0 (Z B _, = 0) 


pr(Z n - 1) - E z „Z n - E Zn i E z l 'iZn-, m z„.,Z n 

= *z„-A Z *-i = 0 B pr(Z B _, = 1) = 0 B {0 B _,pr(Z B _ 2 = 1)} 
= ... = 0 B 0 B _i...0 lP r(Z o = 1) = 0 B ...0 lP r(X o = 1) 


-Mi-®. □ 

Consider an MC {Z B :n = 0,1,2, ...} having time-independent transi¬ 
tion matrix P. If the distribution of the initial state is pr(Z 0 = i) = p. 
(« = 0. 1, 2, ...), then by applying the Markov property to equation 2.23 
we obtain the joint distribution of Z 0 , Z,, ... , Z B : 

pr(Z 0 = i 0 , Z, = »,, .... Z B = ij = P lg P ioli P lll2 ■ ■ ■ Pi n _ t i n (4.1) 

Let us denote pr( * |Z 0 = i) by pr,( • ), then pr^Z, = k) = p lk , and by 
the Markov property and time independence 
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P r ,(Z 2 = ;|Z, =k) = pr(Z 2 =y|Z, = k) = p kJ 
After its first step the particle must be in one of the states of S. Therefore, 
by equation 2.12, 

p!v’ = PL(2 2 = j) = I pr,(Z 2 = j\ Z, = fcjpr/Z, = k) = I p ik p kJ 

keS keS 

In matrix notation the right side is the (i,j) element of PP = P 2 , where P 
is the one-step transition matrix. The two-step transition matrix (pj 2> ) is 
thus P 2 . Similarly (p'J 1 ) = P 3 , and so on. In general 

pI; + b) = z Ptt’p” 

keS 

This is called the Chapman-Kolmogorov equation. In matrix notation 

pm + n pmpn 

We can define P° in a natural way to be I = (<5 lV X the identity matrix, where 
is the Kroneeker delta (= 1 if i = j. = 0 if i ^ j). 

Problem 4.7 Consider an MC {Af„ : n = 0, 1,2,...} with finite state space 
S = {0,1,...,s} and transition matrix P = (p fJ ). Suppose that the MC 
has been thinned according to the matrix T = (y (J ). That is, given X n = j 
and X n+ , = j, we delete X B+ , from the realisation with probability y l} 
and retain it with probability 1— y tJ . This thinning has the Markov 
property. Show that the thinned process {Z n :n = 0,1,2,...}, where 
Z 0 = X 0 is an MC, and find its transition matrix P* = (p*). 


Solution. Each Z k of the thinned process is an Xj of the MC. Therefore 

P^ z k+ i =J \ z o = 'o* z i = •••« z * = **) 

= pr(^ +l =j\ x o = «o- X . t = f i> •••» x n k = h) for some 

sequence n 1 ,n 2 ,...,n k+1 

= P r (^„ k ., = j\ x n k = h) by the Markov property of {X,, } 

= pr(Z k+ , = j | Z k = i k ) 
so {Z k } is an MC. 

We consider a decomposition based on the first step, whether or not it is 
deleted. Let 


Then 



if the first step is deleted 
otherwise 


E cZi — E c 2 * 

^ZilZo^ ^Xi.y.ZilZo*!^ 

— f E E s 2 * 

~~ lL X l \Z 0 = i IL Y\X l = X i .Zo=i^Z i \X l = X i .Y = Y.Zo = i^ 

We pick out the coefficient of i.e. 
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(4.2) 


p* = pr(Z, = j\Z 0 = i) 

= Exi|Zo=i^i r |Xi=Xi.Zo=<F >r ^i =y|-X'i = Af,, l'= Y* Z 0 = 0 

Now 

pr(X, = k|Z 0 = 0= P f * 

and 

^ = P r ^ = 1 l-X'j = k, Z 0 = i) = y ik (4.3) 
Also, by the Markov property, if X x = k, then it is as though the thinned 
process starts again from k. Therefore 

pr(Z, = j\X y = k, 7 = 0, Z 0 = i) = S kJ (the Kronecker delta) 
and 

pr(Z, = j\X 1 = k, Y m 1, Z 0 = 0 = pr(Z, = j\Z 0 = k) = p* 

SO 

P r(Z, = j\X t = k, Y = y, Z 0 = i) = (1 -y)5 kj + yp* (y = 0,1) 
By equation 4.2 

P*j ~ ^’Xt\Zo=i^’r\x,~Xi.zo~M^~^x,j +Y P*i)} 

~ ^Xi|z 0 =i{(^ — "*■ Vix, Px,j) (by 4.3) 

= I +?*?**,} = p,/i-y y )+Z p ik v ik ptj 

keS keS 

= Pij-<Iij+ 1 9ikP* r where Q = (q tJ ) = (p y ) 

keS 

i.e. P* = P-Q + QP* 

i.e. (I-Q)P* = P-Q 

Therefore P* = (I-Q)~ '(P-Q) □ 

4.2 Equilibrium distributions We seek the limiting form of P" as n -» oo, 
if the limit exists. 

Problem 4.8 Find IT = lim P" for a time-independent MC having state 
space S = {0,1}. 

Solution. A realisation of this process will be a sequence of Os and Is. 
The transition matrix P must be of the form 

p -(7 ,!„) »<«./>«'> 

We first consider the special cases in which a or P are 0 or 1. If a = /? = 0 
the particle stays in its initial state so 
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i.e. both states are absorbing. If a = 1 and P = 1, the particle on alternate 
steps goes from state 0 to state 1 and back again, so the MC {Z n } is such 
that Z 2n = Z 0 and Z 2ll+1 = 1 -Z 0 , so there is no limiting distribution. If 
0 < a < 1 and /? = 0, state 1 is absorbing, i.e. when the particle reaches 
state 1 it stays there, so 

-C 0 

Similarly if a = 0 and 0 < /? ^ 1, n = /1 0\. For the case 

\l 0 J 

0 < a, P < 1, consider matrix 

*-(: 3 

where A = 1 - a— p. (1 and A are the eigenvalues of P.) Then 

*-C3 

as n -> oo, since |A| < 1. Now consider matrix 


Then 


-G 7) :> 

-Cr 


Therefore 

P 2 = RAR'RAR 1 = RA(R'R)AR-' = R(AIA)R"‘ = RA 2 R‘, 
and 


I>3 _ D A2D-1D AD-1 _ DA3d-1 


and in general 
P" = RA"R 1 


:)-(!)(; 


J- (V 

*+p\p 


X ;) 

T>> 


P 
P_ 

ct+P a 


In' as n 


oo 


where n 0 = p/(u + p), n 1 = a/(a + /?) forms a probability distribution. 
That is, p)"’ -» 7ij independent of the initial state i, so it. is the equilibrium 
probability of the particle being in state j. □ 

In general, if a limiting matrix II exists, then II = In' where 1 is a 
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column vector of Is and it is the equilibrium distribution of the MC. 
To find it' we do not need to construct a matrix R as above. Since 

pn + 1 __ pup 

by letting n -> oo we obtain 

In' = lit'P 


so 

it' = itP (4.4) 

This is a set of simultaneous equations for it; (i.e. it' is the left eigenvector 
of P corresponding to the eigenvalue 1). 

Problem 4.9 Find the equilibrium distribution of the two-state MC 
having transition matrix /I — a a \ (0 < a, jS < 1). 

V p 1-JV 


Solution. This is, of course. Problem 4.8. We solve it' = n'P for it'. Here 
(«o*i) = (it 0 it 1 )^ a l “p'J = (tt 0 (l —a)+7t, p, it 0 o + nf\ —/?)) 


Therefore 


*o« = n { p 

Since it 0 +it, = 1, we must have it 0 = /J/(a + /?), it, = a/(at + /?) 


□ 


Problem 4.10 In a random walk on states { — 2, —1,0,1,2}, if at time n 
the particle is in state i (i = -1,0,1), then at time n +1 it is equally likely 
to be in i — 1 or i+ 1; and if it is at time n in state —2 or 2, then at time 
n+ 1 it is equally likely to be in —1, 0 or 1. Write down the transition 
matrix of this MC, and find its equilibrium distribution. 


Solution. 


-2 - 

1 

0 

1 

2 

-2 /0 

1 

3 

I 

3 

i 

3 

°\ 

P.-'/i 

0 

1 

2 

0 

°\ 

0 0 

1 

2 

0 

1 

2 

° 

T 

0 

1 

2 

0 

i 

2 \0 

1 

3 

1 

3 

1 

3 

0/ 

7 e wish to solve it' = it'P for 

It' 

= 

2*- 

, it 0 it, it 2 ). Clearly by 


symmetry ir_ 2 = it 2 ,it_, = it,. Then 

*0 = 3*2 + 2*l+2*1+ 3*2 


It 

It 


— —7T 4- 

1 3 /l 2^2 

2 = 2*1 


*0 + 3*2 
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(4.5) 

(4.6) 

(4.7) 


(4.8) 


We have further that 

1 = it.j + ir.j+ito + i^+itj = it 0 + 2it, + 2it 2 

Substituting (4.7) into (4.5) or (4.6) gives 

*o = 3*1 

and substituting this and (4.7) into (4.8) gives it, = 3/13, then 

it' = n 

71 '26 13 13 13 26' l - J 


4.3 Applications 

Problem 4.11 The Ehrenfest model of gas diffusion through a porous 
membrane. Two cells, A and B, contain together 2 a molecules of gas. At 
each time instant a molecule is chosen at random from the 2a, and if the 
chosen molecule is from cell A it is put into cell B, and if it is from cell B it 
is put into cell A. Find the transition matrix for the MC{X n :n = 0,1,2,...}, 
where rv X n is the number of molecules in cell A at time instant n. Deter¬ 
mine a differential-difference equation for 

F n (s) = Ex„|x 0 =i sX " 

for a fixed i. Find the equilibrium distribution of the MC. Determine 

= Ex„|Xo=i-^n - 


TABLE 4.1 




Probability 

Comment 

k 

k + 1 

(2a - k)/2a 

This is the probability of 
selecting a molecule 
from B 

(k = 1,2,...,2a - 1) 

k 

k - 1 

k/2a 

This is the probability 
of selecting a molecule 
from A 

0 

1 

1 

State 0 is a reflecting 
barrier 

2 a 

2a - 1 

1 

State 2 a is a reflecting 
barrier 


Solution. From Table 4.1 of possible transitions, we find that 
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Pij = pr(*„ + . =j\ X n = 0 

(2a — i)/(2a) (j = i+1; i = 0,l,...,2a-l) 
i/(2a) (/ = 1-1; i = 1,2,. ..,2a) 

0 (otherwise). 

We shall write a for 2a. By a decomposition based on the last step 

^1+ l( S ) = ^X„«,|Xo = < S *' ' = * ^X„ ♦ I |X„ * Xn.Xo = l 

— F F ,*»♦* 

- C X„|Xo = i E X„.,|X„-X„ A 


by the Markov property. Now 

oXn + l _ 


V-‘(^)+ S ‘ +, (^) (k = l,2,...,a-l) 


'X„*,|X„ = * J 


s 

s“ -1 


(* = 0) 

(k = a) 


1 


= -(l-s 2 )fcs’‘- 1 +s* + ' (k = 0,1,....a) 


Therefore 


a ^»+i( s ) — s2 ^x„iXo=i'^n sX " 1 + as Ex„|x 0 =i s *" 


= (1 — s 2 ) — F n (s) + as F n (s) 


(4.9) 


Let F n (s) -* F(s) as n -»oo; then 

aF = (1 —s 2 )F' + asF, 
dF 

i.e. »F = (l+s)—. 

as 

Therefore 

F(s) = {c(l +s)}“, where c is a constant. 
But F(l) = El = 1 = (2 cf 
Therefore c = { 

Therefore 

-£o«r-££(;)* 

Therefore 

.« 

t 

i.e. the equilibrium distribution of {X n } is Bin(2a,^). 
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T lf we set s = 1 in equation 4.9 and put M n = F„(l) we obtain the identity 
a = a. However, if we differentiate equation 4.9 we obtain 

«F; +1 (s) = (1 — s 2 )F”(s) +(a — 2)5 F' n (s) +aF n (s) 

Now, if we set s = 1 and a = 2a, we obtain 



2«M„ +l = 

(2a — 2)M n + 2a 

i.e. 

j fl = 


Therefore 

/ 

1\ / 

IV / IV 




. 

i.e. 


/ lV 


M n = fl+| 



Problem 4.12 A simple queue model. A single server at time instants 
1,2,... serves a customer, if any are waiting. Suppose that in time interval 
(n,n+1) X n customers arrive, where {X n } is a sequence of independent 
rvs, each distributed as rvAT, where pr(AT = k) = p k (k = 0, 1, 2, ...). If 
rv Z n is the number of customers present (including the one being served) 
at time instant n, show that {Z n : n = 0,1,2,...} is an MC. Find its transi¬ 
tion matrix and equilibrium distribution. 

Solution. We prove that {Z n } is an MC by showing that Z n+1 depends 
only on Z n and X n and on no earlier information. The number of customers 
present at time n+1 is the number present at time n, less the one served at 
time n, plus the number of new arrivals in (h, n+1), i.e. 


where 

Therefore 


- +1 U. 
= Z . 

-{i 


,-1+X, (Z.- 1.2,...) 

(2„ = 0) 

= Z-A m +X H 

(Z. - 1 . 2 .-) 

(Z„ = 0) 


(4.10) 


Poj = Pr( z , + i =;l z n = 0) = pifX n =;) = pj O' = 0,1,2,...) 
and for i ^ 0 


Pij = P r < z „ + i = J'l z . = 0 = pr( z „ 
= pr(^=/-H-l) = |^- ,+1 


-l + X„=j\Z m = i) 
O' = i— l,f,i+l,...) 
0 = 0,l,...,i—2) 
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i.e. 



0 

1 

2 

3 ... 

0 

/Po 

Pi 

Pi 

p 3 ...' 

p = ' 1 

* 

Pi 

Pi 

p 3 ... 

2 

0 

Po 

Pi 

Pi ••• 

3 ’ 

1° 

0 

Po 

P, ••• 

= ir'P for it' = 

V 

= (*0 

*i n : 

«•••) 

and find 


\ 


k+1 

tt k = 31qPh+ ^ 1tjP k+k _j 0,1,2,...) (4.11) 

These may be solved successively to find it k in terms of it 0 . Then Z n k = 1 
will give the value of n 0 . Alternatively, let us define pgfs 

n(s) = X n k ^, P(s) = Y. Pk^'y 

k=0 k-0 

then P(s) = Es x is known. We call EX = p the traffic intensity (= ratio 
of expected number of arrivals to expected number of departures). We 
multiply equation 4.11 by s* and sum for k = 0 to oo: 

i7(s) = n 0 P(s)+- £ £ n J s J p k+1 _ j ^ +l ~ J 

sk=0j =J 
J 00 00 

= 7t 0 P(s)+- X 1 P,* (by changing the order of 

Sj=i r—k +1 — js*o summation) 


= n 0 P(s)+ ] -{ms)-n 0 }P(s) 

Therefore 


= yi-w 

'' P(s)-S 


Now 


1 = /7(1) = n 0 lim ——-= n 0 lim —- (l’Hopital’s rule) 

TO-> .„ d 5) _ 

as 


|: _(1 -s)P'(s)-P(s) -1 

= 71 r, lim--—-- = 71 


ST1 


Therefore 


F(s)-l ~°p -1 

7T 0 = l-p 
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Therefore 


n(s) = 


(l-p)(l-s)P(s) 

P(s)s 


The condition for stability (i.e. an equilibrium solution) is p < 1, be¬ 
cause if p = 1, then n 0 = 0 so IJ(s) = 0, and if p > 1,7t 0 < 0, and is not a 
probability. If p > 1 the queue size will tend to grow without limit. □ 


Problem 4.13 Suppose that in the simple queue model there is waiting 
room for only m customers including the one being served. Customers who 
arrive to find the waiting room full leave without being served and do 
not return. What is the transition matrix? Show how to find the equilibrium 
distribution. 


Solution. Z n+1 = min (Z n -A n +X n ,m) 

[Z- 1+X n (Z. = 1,2,....m; X n - 0,l,...,ro+l-Z (I ) 

X n (Z. = 0; X„ = 0,1.m-1) 

m (otherwise) 

Therefore 

Poj = P r ( z n + 1 =J l z „ = °) 

'prfX,, = j) = Pj 0 = 0,1,...,m-1) 

pr(X„ Ss m) = p m +p m+ , + ...= P m , say (j = m) 

(otherwise) 


and, for i = 1,2,.... m, 

PiJ = P f C z «+i =)\ z n = 0 



fP 


7+1- 

-0 = 

Pj-i+i 

a 

= i—1 


= - 

P r ( Z n 


m +1 

-«•) = 

= p 

'm+l-l 

O' 

= m) 



^0 





(otherwise) 



0 

1 

2 

... m — 

1 

m 



0 / 

( Po 

Pi 

P 2 

••• p«- 

i 

M 



1 / 

Po 

Pi 

P 2 

••• Pm- 

i 


\ 

P* = 

■ 2 :\ 

0 

Po 

Pi 

”• Pm- 

2 

^-i 




\o 

0 

0 

••• Po 



1 


In solving it*' = n*'P*, the first m equations are the same as in (4.11). 
Therefore n* = cn j (J = 0, l,...,m—1), where c is a constant. This is 
because each it* (j = 0, 1,..., m-1) is the same function of n* as n } is of 
n 0 . The last equation is 


E 
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m- 1 


< = < P m+ .1 *J P m + l-J = K P m + .1 n J P m+ l-J + K P l 

where P m = p m +p m + x + — This reduces,where y m is known by Problem 
4.12, to 


Po n m = O-^iX = c (*o P m+ .2>A+i-j) = cy m say 


Now 


m -1 


Po = P 0 I K j = P 0 c I «,+ <7,n 


j-0 


J=0 


Therefore 


■ p '>/{ p ° ^ "' +y *} 


Problem 4.14 The discrete branching process. Each individual in a popu¬ 
lation gives rise to X new individuals for the next generation, where rv X 
has pgf G(s). Individuals reproduce independently of one another and 
then die. This occurs at the same instant for each individual after a fixed 
constant lifetime. If the process starts with a single individual, show that 
77 n (s), the pgf for Z n , the population size in the nth generation, satisfies 
recurrence relations 


n n+t (s) = n n {G(s)}, n n+l (s) = G{n„(s)} (4.12) 

where clearly I7 0 (s) = s and i7,(s) = G(s). 


Solution. We consider decompositions based on the last and the first 
generations respectively. We can write 

*- + i-*!+•••+**. (4-13) 

since each of the Z n individuals in the nth generation has his own family. 

The family sizes {X ,: i = 1,2,.... Z n } are independently distributed, each 
with pgf G(s). Then, by Problem 2.14, Z n+l has pgf /7 b {G(s)}. 

Alternatively we can consider the populations which grow in the re¬ 
maining n generations from the Z, = X born in the first generation. Then 

Z . + l = Z !; 1, +•••+ z< .* , (4-14) 

where the are independently each distributed like Z n . Then, again by 
Problem 2.14, Z n+ , has pgf G{fl n (s)}. □ 

Making parents die when they give birth is no restriction since we could 
have a parent survive as one of its offspring if G(0) = pr(X = 0) = 0. 
This is because individuals behave independently and alike, so parents 
and offspring will be indistinguishable. On the other hand, if we have 
G(0) > 0, the number of children is not restricted to one or more, so it will 
be possible for the population to die out. We then have the problem of 
determining the chance of extinction. 
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Problem 4.15 Determine M n = EZ n and V n = VZ n for the population 
size in the nth generation. 

Solution. Z 0 = 1 so M 0 = 1 and V 0 = 0. From equation 4.13, 

- Bwr.+ . +x.) - *EX - to 

where a = EX = G'(l) is the mean family size. Therefore 

M„+i = EZ n+ 1 =E Zfl E z „^,|z n =z r ,^ii+i = EJZ.00 = a M n 
This is a recurrence relation, and obviously 

M n = aM n-X = a ( aM „-2) = * 

Also, since the X, are independent, 


= a”Af 0 = a" 


= V(X,+... + **) = kWX = kp 


where P = VX = G"(l) + a — a 2 . Therefore, from equation 2.22, 

V = V7 =FV 74-VF 7 

r ii+l t ^b+ 1 C Z„ T Z„.i|Z„ = Z„^n+l T Y Z n c, Z„,i|Z„ = Z„ i 'ii+l 

= E Zn (2„iS)+V Zn (Z„a) = PM n +«*V n = Poc' + ^V (4.15) 

This is a recurrence relation. If a = 1, then we easily see that V n = np. 
If at ^ 1, we can solve successively with n = 0, 1, 2, .... Alternatively, 
let us use equation 4.14 


il*= 


c Z „ + l = * EZ „ = xM n = 


Therefore 
K+t = vz „ + . = E x v Zn< 


V z Z . = xVZ = xV 

Z n+ i|A=x B+l n n 


r Z «+i + ^x E 2 


= E Jf (A'F)+V x (AV) = FEX + a 2n VX = a 2 "p+<xV n 


(4.16) 


This is another recurrence relation for V n . If a # 1, we solve by subtracting 
(4.15) from (4.16) to obtain 

a(l-a)F = /?a"(l — a") 

i.e. fpn (a = 1) 

K ~ {por-'( 1 -^) (a / 1) 


Problem 4.16 Show that the population process { Z n :n = 0,1,2,...} 
is a Markov chain, and find its transition matrix. If Z 0 = 1, what is the 
chance that the population will eventually become extinct? What if the 
process starts with i individuals? 

Solution. Since 

+ Z z„ ( Z „=U,...) 

(Z n = 0) 

and the X. are independent rvs, each having known pgf G(s), we see that the 
process is an MC. Let 
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, + i jo 


(4.17) 

















PiJ = P r < Z n + l =J\ Z n = 0 

Then clearly 

p 0j = d 0j (the Kronecker delta), i.e. state 0 is absorbing 
p tJ = pr(X = j) = p r say 

But what is p., (i = 2, 3, 4, ....)? Given Z n = i, from (4.17) and Problem 
2.12, 

= f Q Pi/ = E z„. 1 |2--i sZ " = W) 1 

so p.j is the coefficient of s 1 in {G(s)}'. 

If Z 0 = 1, the probability that the population is extinct on or before the 
nth generation is p { " ] 0 = pr(Z n = 01Z 0 = 1) = q n say. Clearly q n increases 
as time passes, that is as n increases. We can assume that 0 < p 0 = p l0 = 
< 7 , ^ 1, since if p 0 = 0 no extinction is possible. The sequence {q n : n = 
1,2,...} is therefore bounded above (by 1) and increasing, and so must have 
a limit q. But by (4.12) 

9„ = = G{i7._ 1 (0)} = G(q n _ 1 ) (4.18) 

Therefore, as n -> oo, limit q satisfies 

q = Giq) (4.19) 

We note that, since G(l) = E x l x = 1, q = 1 is always a solution. There 
may be another solution and, if there is, it is the one we want. To prove 
this, let q* be any root. Then since G(q) is a power series in q with positive 
coefficients (they are probabilities) it increases in 0 < q < 1 as q increases, 
so by (4.18) 

«, = G(0) < Giq*) = q* (4.20) 

and 

q 2 = Giq,) < G(q*) = q* 

(since, by (4.20) q , < q*, so Giq,) < Giq*)). 

By induction, q n < q*. Therefore, as n -* oo 

q = lim q < q* 

n-ao 

and so the chance of extinction, q, is the smallest positive root of q = Giq). 

If the process started with i individuals, the probability of extinction is 
the chance that each of the i lines independently dies out. It is therefore q‘, 
where q is the smallest positive root of q = G(q). □ 

Computationally we have described a simple iterative procedure for 
solving equation 4.19, namely 

^ + i = G(<z„), q , = p 0 

The condition separating the cases q = 1 and q < 1 is given by observ¬ 
ing the slope at s = 1 of the function G(s) (see Figure 4.2). 
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The slope at s = 1 is G'(l) = EX = a, the expected family size; and so 
q j J according to whether at 

4.4 Classification of the states of a Markov chain State j is periodic 
with period d if 

p^> > 0 im = 1,2,...), pj"’ = 0 (n / md) 

i.e. the particle can return to its initial state j only at times d. 2d, — 
The period d is the greatest common divisor of all n for which pj"’ > 0. 
If d = 1, then state j is called /jperiodic. Each state of the unrestricted 
random walk on the integers (Problem 3.1) has period 2 since the particle 
can only return to it in an even number of steps. 

Let rv Tjj be the time at which the particle returns to state j for the first 
time, where T }j = 1 if the particle stays in j for a time unit. The state j is 
recurrent if prfT^ < oo) = 1 and transient if pr(T^ < oo) < 1. 

Let rv T.j be the time for the particle to go from state i to state j for the 
first time. Then T tJ is called the first passage time from i to j. (See Problem 

3.15.) Let T have pgf FJs). If J>/s) = £ p<?s", then 

11*0 

P tJ (s)-6 tJ = F tJ is)P t} (s) (4.21) 

where S tJ is the Kronecker delta. This is the renewal equation. 

Problem 4.17 Derive the renewal equation. 

Solution. Let rv be the time at which the particle starting in state i 
is in state j for the fcth time. Then Mjj’ = T t , and 

M<;> = Ty + TJ‘»+ T]f+... + P)~ i) (k = 2,3,...) 
where each term, by the Markov property, is independently distributed, 
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and, by the time independence of the MC, each Tj 1 "’ is distributed as T jr 
Then, if MJ*’ has pgf G ijk (s), 

G ijk (s) = Fy(s) {FjP)} k ~ 1 = F^Gjj,. ,(s) (k = 1,2,3,.. .). 

Now by (2.1) 

Pi"' = pitMjj' = n or M®’ = n or ...) 

= prfM'j’ = n)+pr(M< 2) = «)+.... 
Multiply by s", and sum over n = 1,2,.... 

P,J (s)-/C = f pS 7S" = G yi (s) + G IJ2 (s) + G ij3 (s) + ... (4.22) 

= Fy(s) (1 + Gjji(s) + Gjj 2 (s ) + ...) 

= F..(s)[l + {P„(s)-p‘°>}] 

by setting i = j in (4.22). Then (4.21) follows by noting that p { ° ] = S tJ . □ 
When i = j and s = 1 we have 

P^(1) = {1-F JJ (1)}- 1 

then, since pr(T ;j < oo) = F^fl), the condition F jV (l) = 1 is equivalent 
to Pjj(l) = oo, and F^(l) < 1 to P JV (1) < oo. Thus state j is recurrent or 
transient according to whether P^(l) = oo or P^fl) < oo. 

Problem 4.18 Determine F y (s) and P tJ (s) for the unrestricted random 
walk of Problem 3.1 for every integer i and j. 

Solution. The solution of Problem 3.4 is that 

P 00 (s) = u(s), P 0 , (s) = {tl(s)-1 }/2 qs (4.23) 

where u(s) = (1 —4pqs 2 )~*. By symmetry we can interchange p and q to 
obtain 

P, 0 ( s ) = {w(s)-l}/2ps 

Starting in state i, the particle cannot be allowed to pass through state 0 
if it is reaching 0 for the first time. State 0 can therefore be treated as an 
absorbing barrier. Then (for i > 0) F, 0 (s) is the pgf for the time to a 
gambler’s ruin if he starts with £i. In Problem 3.11 we saw that 

F i0 (s) = {F, 0 (s)}‘ (« > 0). 

By the stationarity of the random walk (Problem 3.5) 

+jj (^) = P»M. F i+jj(s) = 

From equation 4.21, 

F 00 (s)= l-{P 00 (s)}-> = 1-W5)}" 1 

p i0 (s) = F i0 (s)P 00 (s) = {F J0 (S)}'P 00 (S) (« = 1,2,...) 


and 
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Therefore from the case i = 1 

F 10 (s) = P 10 (s){P 00 (s)}- 1 = [1 -{4s)}- l ]/2ps (4.24) 

The alternative solution of Problem 3.11 is completed by equation 4.24. 
Together the displayed equations from (4.23) to (4.24) enable us to deter¬ 
mine F.j(s) and P u (s) for every i ^ j. Similarly 

F_, 0 (s) = {F- U 0 (s )} 1 

where, by symmetry, F_, 0 (s) is F 10 (s) with p and q interchanged, i.e. 
[1 — («(s)} “ *]/2 qs. Then 

P-U 0 W = P-, 0(^00 (s) = {F_ it0 (s)YP 00 (8) (i = 1,2,...) 

and so we obtain F ij (s) and P (j (s) for every integer i and j. □ 

Problem 4.19 Determine ET f0 and VT i0 for the unrestricted random 
walk. 

Solution. From F 10 (s) by equations 2.24 we find that if p < q 
ET 10 = (q-p)-', VT l0 = 4pq/(q-p) 3 
Therefore, from equation 3.11 

ET |0 = iET l0 = i(q-p)-\ \T i0 = iVT 10 = 4 pqi/(q-p) 3 

If p ^ q, F\( 1) = oo, F''( 1) = oo and so the moments ET i0 and VT f0 do 
not exist. □ 

State i is null-recurrent if i is recurrent and ET„ = oo; state i is positive- 
recurrent if i is recurrent and E T u < oo. 

State i leads to state j (we write: i —► j) if for some integer k ^ 0, p^ > 0. 
The particle then has positive probability of reaching state j from state i. 
States i and j communicate (we write: i<-+j)tf i -+j and j -► i. Now is 
an equivalence relation, so the states may be partitioned into equivalence 
classes, called the irreducible classes of the Markov chain. A Markov 
chain of just one class is called irreducible. Clearly, whilst it is possible for 
a particle to leave an irreducible class to visit another, it can never return, 
otherwise the two irreducible classes would communicate and be a single 
class. The random walk with 2 absorbing barriers has 3 classes. 

Both periodicity and recurrence/transience are class properties. For an 
MC with only finitely many states, at least one state must be recurrent, 
and every recurrent state must be positive recurrent. The reader should 
be able to argue why this is so. 

Kolmogorov's theorem is that if an MC is irreducible and aperiodic, then 
there is a n j such that 

j/ff -+ n. as n -► oo 
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The {uj } is the equilibrium distribution, which by this theorem exists, 
and which we have seen satisfies equation 4.4: 

n' = *'P 

If an MC has an equilibrium distribution it is called ergodic. It then forms a 
single irreducible class, and every state is positive recurrent and aperiodic. 

Problem 4.20 Show that the unrestricted random walk with p = q = \ 
is null-recurrent. 

Solution. P it {s) = P 00 (s) = (l-s 2 ) - * 

Therefore 

/yi) = oo 

so the random walk is recurrent. 

F tl (s)= 1 -{PM' 1 = 1-0-s 2 ) 1 
so 

E T i( = F;,(l) = lims(l—s 2 )-* = oo 

s —♦ 1 

therefore the random walk is null-recurrent. □ 

Problem 4.21 A simple storage model. The following is a simple model of 
storage for a disposable or edible commodity, such as razor blades or 
bottles of wine. At each time instant an item may be consumed. When 
stocks run out, j items (J = 0, 1, 2,...) are purchased with probability r } . 
Consider an irreducible MC with transition probabilities 

Poj = r j O'= 0,1,2,...) 

P,i = P O' = 1,2,...) 

P u -i = q = l~p 0 =1,2,...) 

Find the pgf F i0 (s) for the first passage time T l0 from state i to state 0. 
Show that the MC is positive or null-recurrent according to whether the 
mean of the distribution {rj} is finite or infinite. 

Solution. In state i (>0) the next step is to * — 1 with probability q or no 
change with probability p. Now 

T i0 — + T ( _ t ,_ 2 +...-t-T 10 

where each component is independently distributed like T 10 . We consider 
a decomposition based on the first step, X. Given X = — 1, T l0 = 1 and 

E r,o|x--i sT, ° = s ‘ = and given X = 0, T l0 = 1 + T' 0 where T' 10 

is distributed like T )0 , so 

P c^tO — p cl + ^lO — oIT ( c\ 

^r.ojjr^o 5 E r, 0 s sr io' s > 


Therefore 


f>oO) — Es 10 — Ej.Ej.ij.jS 710 

= pr(X = — l)s+pr(A" = 0)sF 10 (s) = qs+psF l0 {s) 

Therefore 

F 10 (s) = qs/(\-ps) 

Therefore 

F i0 (s) = {F 10 (s)}‘ = {qs/(l-ps)}‘ 

The chain is given to be irreducible. (The sequence jr,} must therefore have 
an infinite number of nonzero elements.) To study recurrence we need 
therefore investigate only one state. It is convenient to examine state 0. 
Given that X = 0,thenT 00 = 1; and given that X = y,thenT 00 = 1 + T, 0 . 
Therefore 

Erooix-js 7 ™ = E ryo s 1+r " = 5{F 10 (s)V 

Therefore 

F 00 (s) = Ex E roo|x-x sTo0 = r oS+°I r j{Fio( s )Y = S Z f j -p$J r J 


= sR{qs/(l-ps)}, where R(s) = £ r.s' 

J =o 1 

Therefore 

F 00 (l) = F(l) = 1 since {r^} is a distribution 


State 0 is therefore recurrent, and so the MC is recurrent. 


Further, F„ 0 (l) = 1 +-/?'(!), where F'(l)is 


the mean of the distribution 


{r,}. F' 0 (l) is infinite or finite according to whether F'(l) is infinite or 
finite. This is the condition for state 0 (and so also the MC) to be null or 
positive recurrent. □ 


Problem 4.22 Each morning, records of the previous day’s business arrive 
at the accounts office to be entered in the ledgers. The office on any day is 
capable of writing up k days’ records with probability p k (k = 0, 1, 2,...). 
Show that, if the office staff work diligently, Z n , the number of days’ 
records at the start of day n waiting to be written up, forms a Markov 
chain, and find its transition matrix. 

Show that the trial solution n } = (1 {j = 1, 2,...) satisfies the 

equations for the equilibrium distribution. What in this equilibrium 
situation is the probability that the previous day’s accounts will be dealt 
with immediately on arrival at the office? 
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Solution. Let X n be the number of days’ records actually dealt with on 
day n. Then the number waiting on day n +1 is the number waiting on day 
n, plus that for day n, less those dealt with on day n, i.e. 

Z.+ t * Z n+'~ X n 

and this depends only on Z„ and an rv X n which depends only on Z„, so 
the process is an MC. Now X n is the number capable of being dealt with if 
at least that number is waiting, otherwise X n is the number waiting. 
Therefore 


pr(X n = j\Z n = i)=< 


O' = 0,1,...,i-l) 


X P k = P, say 0 = 0 

k = l 


j0 (otherwise) 

Therefore 

Pu = P r(Z B+1 =J\Z n = 0 = pr(Z„+l-*„ =j\Z„ = i) 

r P t (i+l-j = i; ie. 7=1) 

Pi+\—j 1 j 0,1 ,...,i 1, 

i.e. j = 2,3,...,/+!) 


= pr(X n = i + l-j\Z n = 0 =_ 


.0 


(otherwise) 


i.e. the transition matrix is 

1 2 3 4 5. 

\ /P l p 0 0 0 0 . 

P = 2 / Pj Pj p 0 0 0 ... 

3 p 3 Pl Pl Po 0 ••• 


The equilibrium equations, n' = n'P, are 
n, = n l P l +n 2 P 2 +n 3 P 3 +... 

n k+ 1 = + Pl "*" 7t li + 2 Pl + - 


(k = 1,2,...) 


Using the trial solution: tij = (1 — AJA-' -1 , we find that X must satisfy 

X = G(X) = £ p k X k 

k-0 

where G(s) is a known pgf. This equation emerged in Problem 4.16 about 
the discrete branching process. We saw there that if G'(l) < 1 then the 
equation has just one root, X = 1. Then n. = 0 and does not form a 
distribution. For equilibrium, therefore, we must have G'(l) > 1, and take 
X to be the smaller root, 2 0 , of X = G(2). 
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The probability of the (n — 1 )th day’s business being entered into the 
ledgers first thing on day n is 

prfZ.., = 1, X n * 0) -► 7tj(l -p 0 ) = (l-2 0 )(l-p 0 ) as n - co 

□ 


Problem 4.23 Consider the library loan process of Problems 1.5 and 4.5. 
If F(t) has a finite mean, show that the MC is irreducible, aperiodic and 
positive recurrent. Find its equilibrium distribution. 


Solution. The MC is irreducible since, for each (i,j), k ■= j +1 is such 
that p ( ( V > 0. The possible path from i to j in /+1 steps is i to 0 on the first 
step, from 0 to 1 on the second, then 1 to 2, and so on, ending on the 
(/+ l)th step with j — 1 to j: 


Py 1 ^ PtoPoiPi 2 "-Pj-i.j > 0 

Aperiodicity and recurrence are class properties so we consider only the 
one state, 0, of the irreducible chain. Since p^ = F(l) > 0, the MC is 
aperiodic. State 0 is recurrent if pr([T +1] < oo) = 1, where [T +1] is 
the number of weeks the book is on loan, i.e. the smallest integer not 
smaller than T, and is the first return time to state 0. Now 
pr([T +1] < oo) ^ pr(T +1 < oo) = F(oo) = 1, so the MC is recurrent. 
Now since E[T +1] $ E(T +1) = ET +1, and ET < oo, we have that 
E[T +1] < oo, so the MC is positive recurrent. 

An irreducible, aperiodic, positive recurrent MC is ergodic and has an 
equilibrium distribution, n', given by n = nT, 


_, _ F(2)—F(l) . _ F(3)—F(2) . 

' "• f(1)+ ”' 1-F(|, + "i 1— F (2) +- 

1-FC/+1) „ , , , 

n J+t ~ n j j -f(j) ^ ~ O’ 1 ’ 2 *---) 

f i -m 1 fi-Fo+D] _ i— f(/+ 
"^-‘{l-FO'-Djl 1-F0) -F{j- 

= ... = n o {l-F0'+l)} 

itj = n 0 {l-FU)} (j = 0,1,2,...) 


Therefore 

Since 


1 = S I {1-^CO}, 

J-0 j=0 


71, = _ 

J °o 


1-F0) 


I {1 — F(k)} 

k = 0 


□ 
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We can easily show that £ {1 — F(k)} = E[T + 1]. 

*<= o 

Problem 4.24 Achilles and the tortoise. Zeno (495-435BC) reported on a 
race between Achilles and a tortoise. Suppose that the tortoise is given 
w time units start. Suppose also that during each time unit it runs a distance 
of a yard or else it stays where it is according to an independent Bernoulli 
trial with probability of moving 0 (0 < 0 < 1), and that Achilles runs at a 
steady rate of one yard each time unit. 

During the initial w time units the tortoise will run X x yards to position 
Z, = X x . While Achilles is reaching Z,, the tortoise will run on a further 
X 2 yards to position Z 2 = X x + X 2 . In general while Achilles is running 
fromZ n _, to Z a , the tortoise is running a further X n+x yards to Z n+1 = 
Z n +X* n + x . 

Show that {X„ :n = 1,2,...} is a Markov chain, and that EX n = w0"; 
hence find EZ n and the mean of Z = lim Z n , the position of the tortoise 

fl-»GO 

when Achilles catches it up. 

Determine the distribution of Z directly. 


Solution. X n+ , depends only on X n (it is as though the race starts again 
with the tortoise having X n time units start), so {X B } is an MC. Clearly 0 is 
an absorbing state, when Achilles catches up with the tortoise. Now X , is 
Bin(w, 9) and X n+1 \X n = k is Bin (k, 0). Therefore 

M n +i = E-^n+i = |X„=X„-^it+l = E x „(X a 0) = 0M n 

Therefore 

M„ = 9M n _ t = 9 2 M n _ 2 = ... = 9 n ~ 1 M l = F-'iwO) = w0" 

(-» 0 as n -*■ oo). 


EZ.-E^+...+*„)= I M k = y3^(l-0") 

fc = 1 

(' 


wd 

T "—0 


as n 


»). 


Directly: the tortoise runs a distance Z before Achilles catches up with it, 
where Z is the number of successes before the wth failure in a sequence 
of independent Bernoulli trials with probability of success 0; therefore Z 
is negative binomial, NB(w, 1 — 0). □ 


EXERCISES 

1. Nuclear chain reactions. Consider a population of neutrons which 
are being bombarded by other particles. The neutrons are inactive unless 
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directly hit by a particle. By fission the particle is then broken into a 
fixed number, m, of the neutrons, and energy is released. Under a steady 
stream of bombarding particles each neutron independently in each short 
unit time interval has a small probability 0 of receiving a direct hit. If the 
number of neutrons initially is mi 0 , show that the number present at the 
nth unit time point is a Markov chain. What is its transition matrix? 

2. By defining the sequence {Xj : j = 0,1,2,...} of Bernoulli rvs, where 
Xj = 1 if Zj = ij and X j = 0 otherwise, use the result of Problem 4.6 
to establish equation 4.1. 

3. The simple queue model. Show directly from equation 4.10 that 
I7 n (s) = E s 2 " satisfies recurrence relation 

sn „+i ( s ) = {(*“ DP r (Z„ = 0) + njs)}P(s) (4.25) 

where P(s) is the pgf for X. 

If 

OO 

[J n (s) -► /7(s) = X n k s* as n oo, 

k = 0 

show directly from EZ n+1 = EZ n -EA n + EX n that n 0 = 1 -p, where 
p = EX. By letting n -**in equation 4.25 find /7(s). 

4. A single-server queue with batch arrivals. Let us call the nth customer 
served C n . Let B n be the number of customers in the first batch to arrive 
after the departure of C n , and A n be the total number of customers who 
arrive while C n is being served. If Z n is the size of the queue at the departure 
of C n , show that 

(Z.- 0) 

Suppose that when Z n = 0, B n is independent of A n+ 2 , the B„ have pgf 
B(s) and the A n have pgf 4(s). Use the method of Exercise 3 to show that 
/7 n (s), the pgf for Z n , satisfies 

*n n+l (s) = [{B(s)-l}pr(Z (I = 0)+n n (s)]A(s) 

Deduce the form of the limiting pgf. 

5. Consider the 2-state Markov chain {X n : n = 0,1,2,...} of Problem 
4.9. Show that the process { Y n : n = 0, 1, 2,.. .}, where 

if(* 2B .*2. + ,) = (0,l) 

=0,0) 

tf(*2„,*2„ +1 ) = (0,0) 

X(X 2n ,X 2n+l ) = (\,l) 
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is an MC, and find its transition matrix. Define the process 
{Z k : k = 0, 1, 2, ...}, where Z k is the sequence constructed by deleting 
values 2 and 3 from the sequence { Y n }. Show that {Z k } is a 2-state MC 
with transition matrix p 00 = p n = A, p 0l = p l0 = 1—A, where 
A = (3-a 

6. /4 branching process. Suppose that every individual at each time 
n = 0,1,2,... becomes a family, where the family sizes are independently 
distributed with pgf G(s). During each unit time interval (n,n+l) immi¬ 
grants arrive in numbers which are independently distributed with pgf 
H(s). Show that 77 n (s), the pgf for the population size just after the nth 
generation, satisfies 

n n+1 is) = H{Gis)}n n {(Hs)} 

7. A branching process. For the discrete branching process of Problem 
4.14, let rv Y k = Z, + ... + Z k be the total number of descendants in the 
first k generations. By a decomposition based on Z,, the number in the 
first generation, show that H k (s), the pgf for Y k satisfies recurrence relation 

H k (s) = 

Deduce a formula for EF k in terms of E F t _, and the mean family size EZ t , 
and use it to find EY k . 

8. Find the classes formed by the states of the Markov chains with the 
following transition matrices. Describe the classes as transient, positive- 
recurrent or null-recurrent. 

(*) P 02 = Pn = Pi.l-i = P|,l +1 ~ 2 ~ 2,3,...), 

p tJ = 0 (otherwise). 

A o * o\ 

• Uo 1 0 

\0 i 0 \j 

9. A random walk with a single impenetrable barrier . Show that the Markov 
chain with states {0,1,2,...} and having transition probabilities 

Poo = Pox ~ Po 

Pi,i-x = » Pi.fH-1 = Pi = 1*2,3,...) 

p.. = 0 (otherwise) 
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where p. + q. = 1 (i = 0, 1, 2_), has an equilibrium distribution if and 

only if the series Ir k converges, where 

r k = (PoPl--P*)/tai‘?2---<?* + |) 

10. The dam storage model. Consider the dam storage model of Problem 

1.6. Suppose that the daily inputs {Y n } are independent, identically 

00 

distributed rvs with pgf G(s) = £ g k s k . Show that {Z n }, where Z B is the 

k = 0 

content after the unit release on day n, is a Markov chain, and find its 
transition matrix. 

If (jr 0 ,jr 1 ,...,jt w _ 1 ) is the equilibrium distribution, prove that 

v k = 7i k /n 0 (k = 0, 1, 2, ..., w — 2) does not depend on w, and that 

00 

Y, v i, s* = G(0)(1 — s)/{G(s)—s}. If the inputs have a geometric distribu¬ 
te) 

bution, what is this equilibrium distribution? 

11. Achilles and the tortoise. For Problem 4.24, show that 

VX n = w0"(l-0"), covfA^A^J = w0" +m (l —0"). 
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Chapter 5 

The Poisson Process 

We now consider stochastic processes in which changes of state occur at 
random time points. First we define the Poisson process {N 0t : t e [0, oo)}, 
which gives the times of these jumps. Let rv N tx be the number of point 
events which occur in time interval (r,t]. If the stochastic process 
{N 0t : t € [0, oo)} is 

(i) time independent, i.e. for each k, pr( N, , +t = k) depends only on t (5.1) 

(ii) has independent increments, and is (5.2) 

(iii) orderly, i.e. pr(N lt+t > 2) = o( t) as t -*• 0 

where, if A(t) = o(t) as t - 0, then A(t)/t -» 0 as t - 0 (5.3) 

then the process is a Poisson process. 

Problem 5.1 Show that {N 0t } is a Poisson process if, for a positive 
constant A, 

(i) P r(W».«+j = 0) = l—XA + o(A) 

= 1) = Ad+o(d) 
pr(N M+4 > 2) - o(A) 

and 

(ii) N l t+A is independent of N ot for all t and A. 

Solution. By (ii) {N 0t } has independent increments. Since the probabilities 
in (i) do not depend on t, {N 0( } is time independent. We are given that 
{ N 0l } is orderly; so {N 0t } is a Poisson process. □ 

Problem 5.2 Prove that, for fixed t, is &(h). 

Solution. We shall abbreviate N 0l to N, here and in the following prob¬ 
lems. Let n t (s) = Es A ', then 

/7, +J (s) = Es n ‘* 4 = Es A,+(N * 4d “ N ' ) 

By (5.2X N, and N l+A -N, are independent and, by (5.1), N l+A -N, is 
distributed a s N d -N 0 = N A (since N 0 = 0). Therefore 

n t+A (s) = n,(s)Es N ' = /7,(s)|f o pr(N. = fc^J 

= /7,(s){(l-Ad)s 0 +/ds 1 +o(d)} by condition (i) of Problem 5.1 
Therefore 

A A 
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Let A -* 0, then 


dI7,(s) ... 

—fi 2 = -A(l-s)/7,(s) 


Therefore 

n,(s) = e -A,<1- * ) 

the pgf for a &(?.t) rv, where we have used /7 0 (s) = Es* 0 = Es° = 1. □ 

Problem 53 The unrestricted random walk in continuous time. Suppose 
that a particle starts at the origin and makes a sequence of independent 
steps: +1 with probability p, —1 with probability q = 1—p at time 
instants T,, T 2 ,..., which occur as a Poisson process having rate para¬ 
meter A (see Figure 5.1). 



What is the distribution of Z t , the position of the particle at time t, 
t e [0, oo)? 

Solution. Essentially we have the bivariate process {Z,,N t :te [0,oo)}. 
The /cth step is a rv 


_ f 1 with probability p 
* } — 1 with probability q 

which has pgf ps+qs~ l . Then 

Z, = X x +...+X Nt , Z, = 0 (ifTV, = 0) 
so, by Problem 2.14, 

G,(s) = Ez'S 2 ' = E Nt (ps+qs~ i y*‘ = exp{-Af(l-ps-<?s -1 )} □ 

Problem 5.4 Determine EZ f , VZ f . 

Solution. By equations 3.4, 

^z t \N t =kZ t — k{p — q)> V Z| j Nf=k Z f = 4kpq 

Therefore, by equation 2.20, 


Ez,Z, — — (p q)E Nt N t — (p — q)At; 


and by equation 2.22 
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V z Z, = E„ t V Zi|Nt=JVt Z ( + V A ,E Zt|Nt=w .Z, 

= 4f*?E Nt N,+ V Nl {(p-q)N,} 

= 4pqA.t+(p—q) 2 At = At □ 

Alternatively, we could have applied equation 2.24 to G, (s). 

Problem 5.5 Find the distribution of rv T, the time to the first event of 
a Poisson process of rate A. 

Solution, pr(T > t) = pr (the first event occurs later than time t) 

= pr(N 0 , = 0) = e - * 

Therefore, pr(T < t) = 1 —e~ A ', which, by equation 2.2, is the distribution 
function of an <?(/) rv. □ 

Problem 5.6 Find the distribution of rv T*, the time between events of 
a Poisson process of rate A. 

Solution, pr(T* > t) = lim pt{N x+Ax+A+ , = 0\N XX+A = 1) 

= lim pr(/V t+Jt+4+l = 0) 

(since N XfX+A and N t+Ax+A+t are independent) 

= pr(lV 0f = 0) 

(by time independence), and so by Problem 5.5, T* is distributed as T 
and is S(A). □ 

Problem 5.7 First passage times for the unrestricted random walk. 
Suppose p < q. Find the distribution of rv T k0 , the first passage time from 
state k to state 0 (see Figure 5.2). 



Solution. As in the discrete time case of Problem 3.11, a first passage 
from state k to state 0 implies successive first passages from k to k— 1, 
from k — 1 to k — 2 ,.... from 1 to 0, i.e. 

T k0 = ^k,k-l +---+ T 10 

where each T J j_ 1 is independently distributed as T, 0 . The characteristic 
function <£ kO (0) for T k0 is therefore {</> lo (0)} k . We condition on the first 
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step which occurs at a random time T, which by Problem 5.5, is an cf(x) rv. 
_ ( T 4- T 00 = T with probability q 
io with probability p 

depending on whether the first jump, X x , was — 1 or +1. Therefore 
•MG) = E T|0 e‘ 9r *« = E Xt E TlolXi . Xi ^o = <jEe IST +pEe iS<T+rjo) 

= #(0)+p^(0)Ee i9T “ 

= m{q+p<t> 2 oW} = 'i'mq+ P {<f> 10 m 2 '\ 

where \p(Q) = A/(A—id) is, by Exercise 14 of Chapter 2, the cf for T. We 
solve this quadratic equation in <f> l0 . keeping 0 constant; and since 

<t >io (°) = i ( for P < q) 

MO) = {l—V0 - 4 p# 2 )}/2p«A 

where ip = tj/{d). □ 

Problem 5.8 We recall from Problem 4.18 that for the discrete time 
random walk having p ^ q, the first passage time pgf from state 1 to 
state 0 was 

F lo 0) = {l-V0-4p«s 2 )}/2ps 

Why might we have expected the relationship <f> l0 (0) = F 1O (^(0))? 

Solution. In discrete time the first step occurs after unit time, so 
Es r = Es 1 = s; in continuous time Ee iflT = □ 

Problem 5.9 Events occur as a Poisson process of rate A. The number of 
events in (0, X), where X is an rv, is an rv Y. Find the pgf for Y in terms of 
the cf for X. 


Solution. E y|z . ( s > ' = e 2,(1 J) . Therefore 
G y (s) = E y s r = E z Ey| X=x s r = E x e -2Jf(1-s> = E x exp[i{Li(l — s)}2f] 

= <£ x {iA(l-s)} □ 

Problem 5.10 Two independent Poisson processes have rates X and v 
respectively. Find the distribution of rv Y y the number of events in the 
first process which occur before the first event of the second process. 


Solution. The time X to the first event of the second process has 


<M0) = v/(v-i0). Therefore, by Problem 5.9, 

Gy (s) = (p x {U(l -s)} =- - -= 1 ~'W+ V ) 

rW * xX K n v+A(l-s) 1-As/(A+v) 


y/U+y) 

1 —s{l —y/(A+ v)} 
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which is the pgf for a #(v/(A + v)) rv (Exercise 10 of Chapter 2). 


EXERCISES 

1. Customers arrive in a Poisson process at the rate of one per minute 
at a newspaper seller’s stand to buy the evening paper. If the seller has 
just 2 copies left and will not receive a supply of the next edition for 5 
minutes, what is the distribution of the number of his disappointed cus¬ 
tomers? 


2. If {N 0t } is a Poisson process having rate parameter A then, since the 
process has independent increments, the numbers of events occurring in 
non-overlapping time intervals are independent and so, by equation 2.18 
and Froblem 2.7, have zero covariance. Also V/V 0l = At. Use these results 
to show that the process {N tt+X : re [0, oo), t fixed} has covariance 
function 


g(s,t) = cov(Af SJ+t , N,, +t ) = j^ (T ^ ^ 


(|f-s| < t) 
(|t-s| > T). 


3. Insurance claims. Claims are made on an insurance company at times 
T,, T 2 , ..., where 0 < T, < T 2 < ..., which are events of a Poisson 
process having rate parameter A. If the claim at time T n is amount X n , show 
that the characteristic function for Z t , the total amount claimed in 
(0, f), is 

exp[—At{ l — tf>(0)}] 

where the X n are independent rvs having cf 0(0). 


4. Stock control. Orders for a certain item are placed according to a 
Poisson process having rate parameter A. The quantities ordered are in¬ 
dependent S(v) rvs. What is the cf for the total demand over an interval 
of length t? 
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Chapter 6 

Markov Chains with Continuous Time Parameters 


6.1 The theory Here we consider a Markov process {Z,: / e [0, oo)} 
havinga discrete state space S, and time-independent transition probabilities 

Pip) = pr(Z t+t = j\Z x = i) (i,j e S; t, t 6 [0,oo)) 

The transition matrix is P(r) = ( p t It )) (ij e S) 

(i) For any f e [0, oo), pJLt) > 0, £ P,p) = 1, 

jeS 

i.e. P(r) is a stochastic matrix. 

(ii) A continuity condition. As t -► 0. p^t) -► 1; then by (i) p t p) -> 0 as 
t -► 0 for i # j\ therefore p^t) -* &ij, 

i.e. P(r) -> I as t -► 0. 

(iii) The Chapman-Kolmogorov equation holds, i.e. 

P(u+ v) = P(u)P(t>) («, V 6 [0, OO)) (6.1) 

If S is finite, then the unique solution of equation 6.1 is 

P(«) = e“* = £ ^ (6.2) 

n = 0 n\ 

where Q is a constant matrix. Given Q we can find P(t), and given P (t) 
we can find Q. Thus Q can be used to ‘represent’ the process. 

If a remainder term A(t) = 0(t) as t 0, then |^4(f)/f | < c, a positive 
constant, as t -*> 0. Then from equation 6.2, as t 0 


i.e. 

so 


P(0 = I + rQ + 0(r 2 ) 

Pip) = &ij + tc lij + 0(t 2 ) 




t 


dt 


1-0 


= p’ u (0) say (6.3) 


since d tJ = p y (0) (by (ii)). If A(t) ~ ct as t -* 0, then A(t)/t -* c as t -* 0. 
Therefore, for i / j, p t p) ~ q (j t as t 0. Thus q tJ t is the probability of a 
jump from i to j in short time interval t; q tJ is termed the transition rate 
from i to j, and Q is called the matrix of transition rates. From equation 
6.3 we can show that if i # j then q ij > 0 and that q ti < 0. We therefore 
set q, - q u > 0. 


Problem 6.1 Let rv 7J be the time spent in state i before a jump from it. 
Prove that T ( is S(q t ). 


Solution. In Problem 5.6 we proved the special case for the Poisson 
process. For fixed f, define 


67 















P n {i) = pr{Z t = i at times t = t/n, 2t/n ,..., (n - 1 )t/n , t \ Z 0 = i} 


= {Pti(^/ W )} n by the Markov property and time independence 
= {1 — q. t/n + 0(t 2 /n 2 )} n -+ e _,<< as n -► oo 

But, as n oo, P„(i) -> pr{Z t = i (0 < t ^ r)} = pr(7; > f). Therefore 

pr(7] ^ t) = 1 — e" fl,f , the df of an £(q.) rv. □ 

If 0 ^ q i < oo, then state i is called stable ; if q. = oo, then i is called 

instantaneous ; and if q { = 0, then i is absorbing. By Problem 6.1 a particle 
instantaneously jumps from an instantaneous state, and never leaves an 
absorbing one. 

The Kolmogorov differential equations relate P(t) to Q. The backward 
equations are 


i.e. 


dP(t) 

dt 


= Qm 


P'ip) = I 

keS 


and are generally valid. The forward equations are 

dP(t) 


dt 


= P(0Q 


i.e. 


P'ip) = I PiPKj 

keS 


and these do not always hold. 


6.2 Applications 

Problem 6.2 The general time-independent birth and death process with 
immigration. Consider the Markov chain {Z,: t e [0, oo)}, where Z, is the 
population size at a time t. If Z t = i (i = 0, 1, 2,...), then in a short time 
interval (r, t+A), Z, increases by one (by a birth or the arrival of an 
immigrant) with probability X,A + o(A), decreases by one (by a death 
or the departure of an emigrant) with probability p t A + o(A), or does not 
change with probability 1—(2, + /^. )A + o(A). Any other changes must 
therefore have probability o(A). Clearly p 0 = 0. What is the matrix Q? 

Find the Kolmogorov forward differential equations, and determine their 
equilibrium solution if it exists. 


Solution. The state space is {0,1,2,...}. Since the transition matrix 


P(d) is 


pr(Z, + d =J \Z, = 0 = 


2,. A + o(A) 

1 —(A,+/i, )A +o(A) 
PtA + o(A) 

Md) 


0 = i+D 
0 = 0 
(/•=«-1) 
(otherwise) 


(6.4) 
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we have that 


0 

1 

Q = 2 
3 


0 

l~K 

Pi 

0 




1 

~P\ ~ 
Pi 
0 


2 

0 

-Pi~P 

Pi 


3 

0 

0 

A, 


4 

0 

0 

0 




Write 

h = *j n J~Pj+i n J+i O' =0,1,2,...) 

Then 

0 =-lo 

o -Ij-x-Ij 0=1,2,...) 

so 

Ij = o 0 ‘= 0 , 1 , 2 ,...) 

Therefore 


71 j+ i 

/ n j = ^-j/Pj+i 

Therefore 

itj = (it j /n j _ 1 )(itj_ l /n j _ 2 )...(n l /it 0 )n 0 

where 

XX X 

= o = i 

J Pjpj-I-Pl 

Then, since Zitj = 

1, 


so 


i = it 0 i v o 

j-0 

”j = Vj/Z v k (j = 0,1,2,...) 

k= 0 


This will be a probability distribution if and only if Ev k converges. 


(6.5) 


Let us abbreviate p t fl) to p ip and use p' tJ to denote {d/dtfp^t). Then, if 
Z f = i, the forward equations are 

p'io = -j-oPio + PiPn 

P'tj = *j-i Pi.j- i-(Pj+P)Pij+Pj+i Pi.j + 1 0 = 1*2,...) 

For the equilibrium solution, let p^t) -» itj as t -> oo, then 

0 = ~*o n o + Pi n i 

0 = (Xj_ i n j _ i Pjftj) (pttj Pj+ i^j+i) 0 = 1,2,—) 


( 6 . 6 ) 


(6.7) 

□ 


Problem 63 The linear birth and death process with immigration. Con¬ 
sider a population process {Z t :te [0, oo)}, where Z t is the size of the 
population at time t. People in the population behave independently of 
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one another. In any short time interval of length d, with probability 
XA +o{A) a member will give birth to another member, with 
probability pA +o(A) the member will die or emigrate, and with probability 
\—(k+n)A + o(A) nothing will happen to the member. In A also, an 
immigrant will join the population with probability otd+o(d). If initially 
the population contains i members, what is the pgf, /7,(s), for Z t ? 

Solution. We consider a decomposition based on the last A. Then 

Z t+A = z t +x i +...+x Zi + w 

where 


W = 


{J 


if there is an immigrant in (t, t+ d), i.e. with probability a A +o(A ) 
otherwise, i.e. with probability 1 - a.A+o(A) 


and 


X, = 


Then 

and 


1 if the /th member gives birth in (t, t+A\ i.e. with probability 
Ad+o(d) 

< — 1 if the /th member dies or emigrates in ( t,t + A), i.e. with 
probability nA +o(A) 

0 otherwise, i.e. with probability 1 —(X + p)A+o(A). 

E s w = ads' +(1 — ad)s°+o(d) = 1 — (a—as)d +o(A) 

Es x l = Ads' + {1 — (A+/i)d}s 0 + /ids -1 +o(d) 

= 1 + {As—(A+//)+/is _l }d + o(d) 


Therefore 

Es *i+...+Xi+ir = (£**•) (Es Xj )..(Es*‘)(Es H ') = (Es*') i (Es"') 

= {(iA+ot)d +o(d)}s + {l — (/A+i7i + a)d+o(d)} + {//id +o(d)}s _1 +o(d) 

from the expansion and collection of terms which are 0(1) and 0(A). 

Therefore 

P r(Z t+A =j\Z, = 0= prfX, +...+X ( + W =j-i) 

(iA+a)d+o(d) (J = i+1) 

1 -(iX + in + a)A+o(A) (j = «j 

ifiA+o(A) (j = i- 1) 

o(d) (otherwise) 

It is clear that we could have written this, or equivalently the Q-matrix, 

straight down. It is of the form (6.4), with A, = iA + a, //, = in, and so 

equations 6.5 are 


P'io = -Wio + PPn 

P'u = {0 ~ 1M +«}?(. ;-1 ~ 0(A + p)+<*}p t j+(j+l )PPi, j + 1 O’ = 1,2,...) 

( 6 . 8 ) 
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A useful device is to set p tJ = 0 for j = .... -2, -1 (clearly true), then 
equations 6.8 hold for j = ..., —1, 0, 1, 2, .... We multiply equations 
6.8 by s J : 

Pij ** = 2 U~ iy _2 Pi.j - 1 + 1 p lJ _ j - (A + n]sjs> ~ l p,j - <xs j p ij 

+H(j+\)s j p ij+i 

We then sum over j = ...,— 1, 0, 1, 2, _ , and abbreviate to II the pgf 

O t (s) = E Zt | Zo „,s ‘ = ^ Pij(t)$i 

j=-co 

Then 

sn , 2 sn > en „ sn 

dt ds ds ds 

ic. ^-+(ks-n)(l-s)~ = a(s— 1)77 (6.9) 


We shall solve this partial differential equation for II. We shall not discuss 
here why the method works. We solve the auxiliary equations 


dt _ ds _ dll 
1 (As—/<)(1—s) — a(l— s)/7 

We shall suppose A # n (Exercise 5 is the case A = p), then 


(X-p)dt 


Xds ds 
Xs—p 1 —s 


= d log 


Xs—p 
1 —s 


Xs—p 

--e ’ = constant 

1 —s 
and 

/a\ / Xds \ _ dn 

\x)[xs-p) ~ n 

Therefore 

(Xs—pf ix n = constant 

Since the partial differential equation is of the first order, it can have only 
one arbitrary constant, so the two constants must be functions of one 
another, i.e. 

(Xs-pf x n = (6.10) 

We determine f from the initial condition: Z 0 = i, i.e. 77 0 (s) = s'. 
(Xs-pf 1 ^ = 

By setting r = (As-/i)/(l — s), so s = (p 4- r)/(A + r), we determine the 
function /(r). This, when substituted into (6.10) gives /7 = /7,(s). □ 
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Alternatively, given Z 0 = i, Z f has the same distribution as Z r * + Y} ,, + 
...+ Y}°, where Z* is the number of immigrants who arrived in (0,0 
and are still present at time t plus their descendants who are still present 
at time f, and 


{ 1 if the ;th of the i original members is still present at time t 
0 otherwise (j - 1,2,..., i) 


Then Z* and each of the T^are mutually independent, and Z* is distri¬ 
buted like Z t given Z 0 = 0. Also, each Y^is a Bernoulli rv with probability 
of success, pr(T > 0 = e - *" (by Problem 5.5) Therefore 

n t (s) = = ^.o^d-e-^+e^y 

We need therefore have considered only the case i = 0. 

Special cases have been given names: 

k = fi = 0: the Poisson process 
k = 0: the immigration-emigration process 
a = p = 0: the linear growth process or the Yule process 
a = 0: the linear birth and death process 


Problem 6.4 Find EZ f for the population process of Problem 6.3. 

Solution. We can find EZ f and VZ, from the pgf Il t (s). However, it is 
not necessary to solve equation 6.9 to determine them. If we differentiate 
equation 6.9 with respect to s, we obtain 

. . ,dn 


s 2 n „ 

8 2 T1 


s) ^ + 

Now 

n,m = 

and 

8 2 n 


dsdt 

if we assume that 



dl 7 
ds 


( 6 . 11 ) 


s= 1 


= 


8t\ 6s 


dl 


lim ~ lim 
dts-i 

Letting s-. 1 in equation 6.11 we thus obtain a first-order ordinary dif¬ 
ferential equation in M,: 

^-t -(n-X)M, = a 
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( 6 . 12 ) 


Then, since M 0 = EZ 0 = i, 
M 


Since 


6 3 n 


6 2 s8t 


8 fd 2 ri 


s= 1 


6t\ ds 2 


■ - D < 6I3 > 

) 


= -£e{Z,(Z,-1)} =l(V l + M 2 -M l ) 


dV dM 


where V t = VZ r we can differentiate a tidied-up equation 6.11 with respect 
to s, and let s -> 1, to obtain a first-order ordinary differential equation 
for V r into which we can substitute for M t from equation 6.13. Note that 
V 0 = VZ 0 = 0. 

Problem 6.5 The single-server queue with Poisson arrivals and exponen¬ 
tial service times. Customers arrive in a Poisson process of rate A, and 
join the queue. The service times for customers are independent S(p) rvs. 
Determine a partial differential equation for Tl t (s\ the pgf for Z f , the number 
of customers in the queue (including the one being served, if any) at time t. 
Find the equilibrium distribution of the queue size. 

Solution. In (t, t + A\ 

pr(a new customer arrives) = kA +o(A) 
pr(a customer finishes being served and he leaves) = pA + o(A) 
pr(no customer arrives and none completes being served) 

= l-(A + /i)d+o(d) 
pr(anything else) = o(A) 

Let p k (t) = pr(Z f = k\ and abbreviate it to p k . The forward equations are 
Po = --IPo+PPi 

p'k = IPk -1 - (p+2)p k + np k + , (k = 1,2,...) 

Multiply the /cth equation by s* and sum over k = 0,1,2,...; abbreviate 
IJ t (s) to IJ: then 

~ = As/7— n(Tl —p 0 )—A/7+j (77 - p 0 ) = (^) {(As-p)/7 + pp 0 } 

00 

If P*(f) -* Ti k as l -* oo, then 77 -» £ n k s k , and {nj is given by setting 

Ac = 0 

sn n ^ , 

— = 0. Therefore 
8t 
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Therefore 


£ **** = 
k = 0 



where 


n k = (l-p)p k (k = 0,1,2,...) 


P 


p a -1 


E(service time)/E(arrival interval) 


is the traffic intensity. Then n k is a probability only if p < 1. For p < 1 
the sequence {rcj is the geometric distribution &(p). □ 

Alternatively, equation 6.7 gives this result with kj = A, = p. 

This queue is in fact a random walk in continuous time with steps 
+ 1, —1, having probabilities k/(k+p\ p/(k+p) respectively, and steps 
taking place as a Poisson process of rate k+p. There is an impenetrable 
barrier at the origin at which the particle stays for a time which is an 
<?(A) rv and then jumps to +1. 


EXERCISES 

1 Accident proneness. Suppose that if no previous accidents have occurred 
the probability of an accident in (t, t + A) is olA + 0 (A\ but that it is PA + o (A) 
otherwise, regardless of how many earlier accidents there have been. 
If /? < a, then the model represents the case in which the lesson has been 
learnt, but if 0 > a then we have the case in which the shock leads to a 
state in which more blunders are likely. Find the Kolmogorov forward 
differential equations for p k (t) = pr {N, = k), where N, is the number of 
accidents in (0, t), and show that /7,(s), the pgf for N t , satisfies 

en 

— +p(l-s)n = 0S-a)(l-s)e-«' 

Find EN, from this equation (by the method of Problem 6.4). Note that 
this model reduces to the Poisson process when a = p. 

2. Suppose that the probability of an accident in (f,t + d) given that 
k accidents have already occurred is (a+kk)A + o(A) (k = 0,1,2,...). 
Find the Kolmogorov forward differential equations for Pj(t) in this case. 
Verify that they are the same as for a linear birth process with immigra¬ 
tion, the birth rate per individual being k and the immigration rate being a. 
Solve the equations to show that N, has a negative binomial distribution. 

3. Race relations. For the linear birth process with immigration of 
Exercise 2, show that if there are n individuals present at time zero, then 
the population size at time t has pgf 
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s"(e A '—sfe 2 '-1)} " ttix 

If the individuals present at time zero and their descendants are called 
natives, and those who enter after time zero and their descendants are 
called immigrants, prove that if nk > a the expected number of natives 
always exceeds the expected number of immigrants. 

4. The linear birth and death process. If there is no immigration and if 
the birth rate per individual is k and the death rate per individual is p (A # p), 
show that the population size at time t, Z,, has pgf (a-b*p)/(a- op), if 
initially Z 0 = 1, where a = ks-p, b = (s-l)p, c = (s-1)2, \p = 
ex p{(A — p)t}. Determine EZ ( , VZ, and the probability of extinction. 

5. Do Exercise 4 and Problem 6.3 for the cases in which k = p. Check 
that the results could have been derived by setting p = A+e in the solu¬ 
tions for A # p, and letting e -* 0. 

6. Find VZ, for the population process of Problem 6.3, using the method 
outlined following the solution to Problem 6.4. 

7. Consider a community in which there is competition for food. If the 
community contains i (i = 0,1,2,..., n) members at time t, then, during a 
short interval (t, t + A), a new member joins with probability (n - i)kA+o(A); 
each member independently leaves with probability kA+o(A), and there 
is no change in membership with probability 1 -nkA+o(A). Set out the 
Kolmogorov forward differential equations for the distribution of the 
size, Z,, of the community at time t. 

If initially the community had no members, show that Z, is Bin 
(n,Kl-e" 2A ‘)). 

8. The incoming traffic to a car park with n spaces is a Poisson process 
of rate A, but only while spaces remain unfilled. In any short time 
interval of length A each vehicle already in the car park independently 
leaves with probability vd + o(A). Write down the Kolmogorov forward 
differential equations for the distribution of the number, Z f , of spaces 
filled at time t. 

Find the equilibrium distribution of Z r If the car park is large enough 
that there is always room for incoming traffic, and if initially k spaces 
are filled, find EZ r 

9. A single-server queue with constant service times . Customers arrive in a 
Poisson process of rate A, and the service time is unity. Let rv Z f be the 
number of waiting customers, including the one being served, at time t. 
Show that if Y t customers arrive in time interval (t,r+l) then 
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_ jZ,-l + Y t (Z, = 1,2,...) 
t+1 U (Z, = 0) 

Find recurrence relations for p k (t) = pr(Z, = k), and show that if G,(s) 
is the pgf for Z,, then 

s G t+I (s) = {(s-l)p 0 (t)+G,(s)} exp{;.(s-l)} 

(See Problem 4.12 and Exercises 3 and 4 of Chapter 4.) 


Chapter 7 

Non-Markov Processes in Continuous Time with 
Discrete State Spaces 

We now consider population processes for which the lifetimes of members 
are not necessarily exponential rvs. The remaining lifetime of an individual 
will depend on how old he is now, and so the future size of the population 
will depend not only on its present size, but also on the present ages of its 
members. The process is still Markovian if we include this information, 
but we would not wish to keep records on every individual throughout 
their lives. There is no change in the population size except when an 
event occurs: when a child is bom, or someone dies, or an immigrant 
arrives or an emigrant departs. We consider, therefore, decompositions 
based on the times of the first or last of these events. 

7.1 Renewal theory We consider first the special case of a renewal 
process. The function of a single component—for example, a continuously 
burning light bulb—is observed and it is replaced by another immediately 
it fails. The components will have lifetimes which are independent rvs 
{X ( }, each distributed like a positive rv X. Then the rv Z„, the time to the 
failure of the nth component if the first is fitted at time 0, is given by the 
random walk 

Z, = *!+...- \-X nt Z 0 = 0 

See Figure 7.1. We take a fixed time t and are interested in the following 
rvs: 

the number of renewals in (0, r], N, = max{n : Z„ < t} 
the forward recurrence time, T ( + = Z Ni+l —t 
and the backward recurrence time, T~ = t — Z Nt 



—*-*—r, + — 

Figure 7.1 


Now, for a fixed r, 

+T* = Z N + , — Z Nt = X Nt+l 

is not distributed as X, since in any realisation of the process the fixed 
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t is more likely to be in a long interval than in a short one. This makes the 
process non-Markov since T, + depends on T~. For example, if a light 
bulb can be of two qualities—a defective one which lasts generally less 
than a week or a good one which lasts about a year—then, if a bulb has 
already survived 6 months, it is of proven quality with about another 6 
months’ lifetime. 

There is an important and fundamental relationship between the 
counting process { N,: t e(0, oo)} and the random walk process [Z„\ 
n = 0,1, 2,. ..}. The events ‘the number of renewals in time t is at least n 
and ‘the time to the nth renewal does not exceed t’ are the same, and so 
have the same probability. Therefore 

pr(N, 2* n) = pr(Z„ sj t) (7.1) 

Suppose that EX = p, VX = cr 2 . We can approximate the distribution of 
N , for large t by working directly with Z„; that is, 

pr(N, < n)= 1-pr(Z„ t) = l-pr(T < (t-np)/oJn) 

where, by the central limit theorem, rv Y is approximately N(0,1). By the 
law of large numbers n~ 1 Z n -» p as n -> oo, and from this we can show 
that N, ~ t/p as t -*■ oo. 

Suppose that X is a continuous rv with density/(x) (0 < x < oo). 
Problem 7.1 Determine an integral equation for the pgf, /7,(s), for N t . 

Solution. We use a decomposition based on the time U of the first re¬ 
newal. If U > t, then N, = 0; if U = u < t, then N, = 1 +N’,_ U , where 
N' t _ u is the number of further renewals in the remaining t - u up to t, and so 
is distributed like N,_ m . We are using the fact that the Markov property 
holds at renewal points, since then it is as though the process were starting 
off again. Now pr{l/ 6 (u, u+du)} = f(u)du. Therefore 

Es N ' = = J o <K /(u)duE JVt|U=B s w * 

= [ ( '/(u)duEs‘ + *"“+ f” f{u)d U Es° 
i.e. n,(s) = s j' o mn,_ u {s) du +j” /(«) du (7.2) □ 

Problem 7.2 Determine integral equations for the renewal Junction 
H, = E N,, and the renewal density h, = (d/dt) H,. 

Solution. We differentiate equation 7.2 with respect to s, and let s -» 1. 
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Then 


- r o /w,.jw*+. j 


Therefore 

H , = J ' 0 f(u)du+ \' o f(u)H,_ u du 
We differentiate this with respect to t: 

h > = f(t)+fU)H 0 + \' o f(u)h t _ u du 

= Jo rfw (7.3) 

since H 0 = EN 0 = E0 = 0. n 

Since N, ~ t/p as t - oo, H, ~ t/p and h, ~ l/p. 

Problem 7.3 Show that h t is the instantaneous transition rate defined by 
pr{a renewal occurs in (t, t + d) | a renewal occurred at 0} 

= h,A+o(A) 

Solution. If we write equation 7.3 in the form 

h,A+o(A) = {f(t)A+o(A)}+ \' 0 {f(u)du}{h,_ u A+o(A)} 
then the right side is 

pr{the first renewal is in (t,t + A) or it is in (u, u+du) for some 
u e (0, t) and there is a renewal in an interval of length A after a 
further t—u} 

i.e. it is the probability that there is a renewal in (r, t + A) given there 
was a renewal at 0. rj 

Problem 7.4 When the lifetime rv is <£(2), that is, exponential with para¬ 
meter show that rv N, is &>(?.t), that is, Poisson with mean kt, and that 
h t is L 

Solution. We use the definition of h, given by Problem 7.3. If h, is JL, 

then by Problem 5.2 N, is iP(/.t), and by Problem 5.6 the intervals between 
events are <?(2). 

The other way round: we put 

f(u) = Ae" 2 " (0 < u < oo) 
into equation 72. Then 

77,(s) = s | o Ae _A “/7,_ u (s) du+ |°° 2e -2u du 


G 
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We multiply by e**, set v = t — u in the first integral, and write 

A, = e A '/7,(s) 

keeping s fixed. Then 

A t = As A v dv+l 
We now differentiate with respect to t 


— = ksA 
dt ' 


Therefore 

Therefore 


A, = e Aa , since A 0 = /7 0 (s) = Es* 0 = 1 


/7,(s) = e -A M ( = 

the pgf of the 0*(At) distribution. Then H, = EN, = At, so h t = H’ t = A. □ 

Problem 7.5 Find the distribution of the forward recurrence time T r + 
of a renewal process, and the limit of this distribution as t -*• oo. 

Solution. We use the method of Problem 7.3 to find the density g* (x) 
of T* by a decomposition based on the last renewal before t. 
pr{T, + e(x,x+J)} 

= pr{the first renewal is in (t+x,f+x + zl)(x > 0), or the last 
renewal before t is in (t>, u+du)(0 < v < t) for some v e (0, t), and 
the next renewal is in (f + x,t+x + d) i.e. in (t+x-u, t+x-o+d) 

later} 

i.e. g+(x)A+o(A) = {/(f+x)d +o(A)} + [' h, dv {f(t+x-v)A +o(A)}. 

J o 

We set u = t—u in the integral, divide by A, and let A -> 0. Then 

0, + to = /(*+*)+ J' o h,_J(u+x) du (7.4) 

We must suppose that /(x) -» 0 as x -♦ oo. Equation 7.4 is true for fixed t. 
Let t -* oo, then h,_ u — ► l , so 

0i + (*) -*• 9(x) = J o 1 f{ u +x) du = - f(w) dw = -{1 -F(x)} □ 


Problem 7.6 Find the distribution of the backward recurrence time 
T“, and its limit as t -+ 00 . 

Solution. 

pr{77 e(x,x + d)} 
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= pr{there is no renewal in (0, t), in which case T~ = t, or there is a 
renewal in (t — x,t—x + d) (0 < x < t) and there are no more 

renewals in (t—x,t)} 

Nowpr{norenewalin(u, u)| a renewal at u} = pr(Af > v — u) = 1 — F(v — u). 
Therefore 

pr(T, = t) = pr(X > t) = 1 — F(t) 
gr(x) = h t _ x {l-F(x)} (0 < X < t) 

That is, the distribution of T~ is an atom of probability of amount 
1 - F(f )at t and a density (x) over (0, t). As t -*• co,F(t) -<• land h t _ x -» n~\ 
therefore the distribution of T~ tends to a density g(x) =■ {1 -F(x)}/n 
(0 < x < oo), the same limiting distribution as that of T ( + . □ 

7.2 Population processes If instead of replacing a failed component by 
another we replace it with X components, where X is an rv having pgf 
G(s), then the situation is that of the family tree described in Problem 4.14, 
except that the lifetimes of individuals are independent rvs each having 
density /(x) (0 < x < oo). 

Problem 7.7 The continuous-time branching process. Suppose that at 
time 0 the process starts with the birth of a single individual, the founder. 
Find an integral equation for the pgf, /7,(s), of the size, Z,, at time t, of the 
population. 

Solution. We consider a decomposition based on the time, U, of the 
founder’s death. Let N be the number of offspring left by the founder. 
Then E^s* = G(s). If U = u, then, for t < u, Z t = Z 0 = 1; but for 
t > u 

Z = f Z ‘-. + -- +Z ‘-u W-U2,...) 

1 [0 (N = 0) 

where each Z t °2 u is independently distributed like Z,_ u . Therefore, for 
t > u, by Problem 2.14, 

E z,\u=u<t sZ ‘ = G{n,_ u (s)} 

Therefore 

/7,(s) = E Z( s z ' = E v E Zl \v =v s / ' = | o f(u) du E Zt([ ,, u s 7 ' 

= j' o flu) du G{n,_ u (s)} du+ j* flu) du Es 1 

= s{l-F(t)}+ j' o /(u)G{J7,_.(s)}dtt □ (7.5) 

Problem 7.8 Determine /7,(s) in the special case in which the number of 
offspring is always 2 and the lifetime rv is <?(A). 











Solution. After an exponential interval each branch splits into two new 
branches, so G(s) = s 2 and f{u) = Ae”*“ (0 < u < oo). This is the linear 
growth process , a special case of Problem 6.3. Equation 7.5 becomes 

n,(s) = se-*+ J' o Ae-' l “{77 I _ 1> (s)} 2 du 

Multiply by e 2 *, set v = t—u in the integral, differentiate with respect 
to f, divide out e 2 *, and abbreviate /7,(s) to /7. Then 


— = 


i.e. 

, . dn ( 1 l\ JrT „ 1/7 — 11 

17(17-1) = U-1 n ) dn ~ dl ° e n 


so 

1 I 77 - 1 ! , 

log 1 — — — - = ).t +constant 


Now 

— Al 

n 0 {s) - s, so n,(s) = 1 _ (l e -^ )s , 


i.e. Z t - 

1 is ^(e" Al )> a geometrically distributed rv. 

□ 


7.3 Queuing theory 

Problem 7.9 Waiting-time distributions. Suppose that the time intervals 
between the arrivals of customers are independently and identically 
distributed continuous rvs, that the service times of customers are inde¬ 
pendently and identically distributed rvs, and that there is just a single 
server. Label the kth customer C k . Denote by Y k the time between the 
arrivals of C k and C k+ ,; by X k the service time of C k ; and by W k the waiting 
time (which excludes the time in which he is being served) of C k . If the density 
of the rv V k = X k — Y k is g(v) ( — co<v<co), and the distribution func¬ 
tion of W k is F k (w), show that 

F k +i( w ) = f" F k( w ~ v)9(v) dv 

Solution. Assume that C, arrives at t = 0 and finds no one ahead of 
him; therefore W t = 0. Customer C k is in the queue for a time W k + X k . 
If Y k > W k + X k , then = 0; that is, if C^, arrives late enough 
he has no waiting. If Y k W; + X k , then H^ +1 = W k + X k -Y k = W k +V k . 
That is, W k+ , = max(H;+K k ,0). Now F k (w) = 0 (w < 0, for all fc). There¬ 
fore, for w $s 0, 
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F k+i( w ) = P^^+i w) = pr{max(iy k -|- V k ,0) ^ w} = pr^+i; ^ w) 
= pr{ V k e(v,v+dv)}pr(W k < w-u| V k = v) 

= j* r g(v)dv F k (w- v) 

since pr(W^ < tv— v \ V k == t>) = 0 (t> > w) and using the independence of 
W k and V k , □ 

As k -» oo, F k (w) -» a limit F(w) which satisfies 

F(w) = f F(w—v)g(v) dv. 

J ~ 00 

n 

It can be shown that F(w ) = pr( X K < w for all n) (w Ss 0); and if 

k= I 

EV k ^ 0 then F(w) = 0, i.e. the queue grows without limit; but if EK* < 0, 
then F{w) -* 1 as tv -+ oo so the limit F(w) is a df. 

Problem 7.10 The busy period of a single-server queue with Poisson 
arrivals. Suppose that customers join the queue at times which form a 
Poisson process of rate a, and that the service times are independent rvs 
having characteristic function i f/(0). Find equations for the pgf G(s) for N, 
the number of customers served during a busy period, and for the cf 
<p{0) for the busy period, B. 

Solution. The starting times of the busy periods—that is, the time points 
when a customer arrives and can be served without waiting—are renewal 
points. The ends of the busy periods—that is, the time points when the 
server becomes idle—are also. The busy periods are therefore indepen¬ 
dently and identically distributed rvs. 

The number, A, of customers who arrive during the service time, X, 
of the first customer, C t , is ^(ctX), so 

F a\xS = exp{ — ax(l — s)} 

These customers are C 2 . C 3 . ..., C A + Without losing generality we 
impose a last-come first-served queue discipline. Then C A + , is served second. 
While C A+l is being served, other customers arrive and will all be served 
before C A . The number served starting with C A + l before C A is reached 
is a rv N A distributed exactly like N, the number served in a busy period. 
Similarly, the number, N A _ V served, starting with C A , before C A _ , is 
reached is also distributed as N, and is independent of N A . Therefore, 
given A, 

N = 1 +N l + N 2 +...+N A 
where the 1 is for C,’s service, so 

^N|y| = a S> = s { Gfs)}” 
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Therefore 


G(s) — EflS* — E x AN s N ” E yl | jr=Jlf {G(s)}' 4 

= sE*exp[ — aX{\ — G(s)}] = s^[ia{l — G(s)}] 

If N = n, then B = X x 4-... + X n , so, by equation 2.28, 

m = G{m} = m'l'iw-mn □ 

This problem can be formulated as a random walk and as such, a special 
case was solved in Problems 5.7 and 5.8. 

EXERCISES 

1. A paralysable counter. A counter records the events of a Poisson 
process having parameter A. After recording an event, the counter becomes 
locked for a fixed period y, and will miss any events of the Poisson process 
which occur during that time. The intervals between recorded events are 
therefore independently distributed like an rv X +y, where X is an g(A) rv. 
By writing the time to the nth recorded event as 

= (X l + y)+(X 2 +y)+...+(X n +y) 

and using Exercises 15 and 16 of Chapter 2, and equation 7.1, prove that 
N r the number of recorded events in (0, t) if an event was recorded at time 0 
has distribution function 

e -A(!-ny) "y 1 {^(£ — ny)} k 
kk k\ 

2. Suppose that the locked periods of the paralysable counter are inde¬ 
pendent <£(p) rvs. If the counter is initially unlocked and set to record the 
events of a Poisson process having parameter A, show by a decomposition 
based on the final A, that p r the probability that the counter is locked at 
time t satisfies 

^+(p+;.)p, = a 

and obtain the solution. 

3. Find the solution, /7,(s), of equation 7.5 in the special case in which 
the number of offspring of an individual is equally likely to be 0 or 2, 
and the lifetime distribution is <?(2a). What is the chance that the population 
is extinct by time f, if it starts at time 0 with the birth of a single individual? 

4. The linear birth and death process with immigration. Show by a decom¬ 
position based on the arrival of the first immigrant that the pgf, <P t (s), 
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for the population size at time t, if initially there are no members of the 
population, satisfies 

<P,(s) = e " "+1' n t _ u (s)<P,_ u (s)ae “ “ du 

= 0 

where the immigration rate is a, and /7,(s) is the pgf for the process with 
no immigration from a single member given in Exercise 4 of Chapter 6. 
Solve this equation for <P t (s) and verify that the solution is that derived in 
Problem 6.3. 

Show that as r -► oo, if k < p, the distribution of the population size 
becomes negative binomial with parameters a/2 and 1 —k/p. 

5. Consider the linear birth process with immigration, with n individuals 
present at time zero (Exercise 3 of Chapter 6). We can write the population 
size at time t, Z t . in the form 

z, = y‘ 1, + ...+y;"»+x- ( 

where the Y\ J) are independent pure birth processes from single individuals, 
independent of X,, which is a linear birth process with immigration having 
no individuals present initially. Then the pgf for Y is found by taking 
/(«) to be an exponential density in Problem 7.7. Then, by a decomposition 
based on the time of arrival of the first immigrant, find the pgf for X,. 
Hence find the pgf for Z,. 

6. A secret society was started at time 0 by a single individual, the founder. 
In any short time interval (t, t + A) each member independently introduces 
a new member with probability kd+o(A). Members independently 
resign or die at a time after joining which is S(p). The founder, who never 
resigns, dies after the society has been running for a time which is S(k—p), 
where k > p. Derive, by a decomposition based on the admission of the 
first new member, the pgf for the membership size at time t, conditional 
on the founder still being alive then. Hence find the pgf for the member¬ 
ship size just after the founder’s death. 

7. For the model of accident proneness described in Exercise 1 of 
Chapter 6, by means of a decomposition based on the time of the first 
accident, show that the pgf for the number of accidents in (0, f) is 

~ s)l —(/}—a)(l -s)e-‘TOs-(i8-a)}. 

Note that this reduces to the pgf for a Poisson rv when a = /?. 
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Chapter 8 
Diffusion Processes 

These are Markov processes {Z f : f e T} having continuous time para¬ 
meter spaces and continuous state spaces, and for which a small change in 
t results in only a small change in Z,. A realisation can be thought of as 
being the path of a particle moving very erratically in a continuous medium, 
its progress depending only on its current position. 

An important class of diffusion processes are the Gaussian processes. 
If {Z t : t e T} is a process whose state space is the real line, R, and if its 
parameter space, T, is any subset of R, then it is a Gaussian process if, for 
every finite n, (Z ( ,...,Z ( J has a multivariate normal distribution. A 
Gaussian process is stationary if its covariance function g(s, t) = g(t—s). 
The Wiener (or Brownian motion) process with parameter A, {Z t : t e R}, 
is the Gaussian process with independent increments having EZ ( = 0 and 
g(s,t) = Amin {s,t), (A > 0). 

Problem 8.1 If {Z ( } is Wiener with parameter a, show that {l'} = 
{e _2, Z c2A ,} is a stationary Gaussian process, and determine its covariance 
function, g(s, t). 

Solution. If Z, is a normally distributed rv, then so is a^Z^y. where a and 
b are real functions of t, since, for fixed t. a(t) and b(t) are real constants. 
The process {y,} is therefore Gaussian, since Z, is normal. Now 

EY, = e-*EZ elAt = 0 
since EZ ( = 0 for all t. Therefore 

g(t,t + x) = cov(i;, Y t+t ) = E(T, Y,+ t ) 

= E {c-»Z' iM e-* + «Z tlM ,. t) } 

= e-^'^co v{Z e2A „Z e2A(t+ „} 

= ae~ M2,+ r, min {e 22 ', e 22(l+r) } = ae -2 ^ 

which does not depend on t. □ 

These processes can arise as the limit of a random walk. Let us consider 
again the unrestricted random walk, but suppose that the particle takes 
independent short steps of length Ax, to the right with probability p, 
and to the left with probability q = 1 — p, after short time intervals of 
length At. It is a delicate matter how we let both Ax and At tend to zero. 
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During a time interval of length t, the displacement Z, is approximately 
t/At independent steps. Therefore, from equations 3.4, 

Ez , ~ ^(p-<?)Ax = {p-q)t~ 


VZ, % j^4pq{Ax) 2 = Apqt 


(Ax) 2 

At 


If we let Ax and At -► 0 in such a way that (Ax) 2 /At -» a and 


P = 






where a and p are constants, then EZ ( -* pt and VZ ( -♦ at. By the central 
limit theorem (Z, — Pt)/y/(<xt) -» an N(0,1) rv. 


Problem 8.2 Show that u x l , where 

u x , dx = pr{Z, e (x,x+dx)} 


satisfies the forward partial differential equation 


ou 

ft 


du , d 2 u 


( 8 . 1 ) 


Solution. From equation 2.12, by a decomposition based on the last At. 

P r ( Z t +ai = *) = P r ( z , = x-Ax)pr(Z I+A( = x | Z, = x-Ax) 

+ pr(Z, = x + Ax)pr(Z, +4 , = x|Z, = x + Ax) 


Therefore 

U x., + A, = + + 

We expand each term in a Taylor series up to O(At) 2 or O(Ax) 3 , and 
abbreviate u x l to u. 

«x, + a, = « + (At)| + 0(At) 2 

«x-Ax.r = « + (-Ax)^+i(-Ax) 2 ^-|-0(Ax) 3 

«x + Ax., = M + (Ax)|^ + §(Ax)|^ + 0(Ax) 3 
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Then 


P »§ 

«+(Af)—+0(At) 2 


= u(p + q)+(-p+q)( Ax)^ +i0>+9)(Ax) 2 ^+O(Ax) 3 

Now p + q = 1, —p+q = — (/?/a)Ax, therefore 


du 

—+ O(Af) 



Letting A* and At -»0 in such a way that (Ax) 2 /Af -*• a, we obtain 
equation 8.1. q 


Problem 8.3 Show that the Brownian motion process satisfies equation 

8 . 1 . 


Solution. Here EZ, = 0, so p = 0; and VZ, = y. min(t, t) = at. There¬ 
fore. since the displacement Z, is NfO.oct), from equation 2.29 we have 
that 




P,(x,y) dy = pr{Z, s (y,y+dy)\Z 0 = x} 
1 


V(2ttat) 


exp 


{-h^} 


dy (-00 < y < oo) 


It is easy to check that u yt = p,(x,y) satisfies the standard diffusion 
equation 


du . d 2 u 
dt ~ 2 *df 


□ 


Often a and p will depend on x, but not on t. The forward equation, 
which is then known as the Fokker-Planck equation, is 

du d Id 2 

-= -~{P(x)u}-i--— 2 { a ix)u} 

If, for a Markov chain {Z n }, we evaluate the functions 

-Z.) «'<*> - -Z.) 

then we can easily approximate the chain by the corresponding diffusion 
process having p(x) = p*(x) and a(x) = a*(x). 


Problem 8.4 The Ehrenfest model of gas diffusion. Find the diffusion 
limit of the Markov chain of Problem 4.11 if Ax, At ->0 and a -»• oo in 
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such a way that 


(Ax) 2 /At y, a At p 1 

Solution. Let X* = X n —a and k = i — a. If X* = X= k Ax = x, 
say, then 

(fc+l)Ax = x + Ax with probability 

y* y* _• 2~lV a ~ 2(1—X/aAx) 

(k— l)Ax = x —Ax with probability 

j + \k/a = ^1+x/aAx) 

Then 

“x.r+A» = U x-Sx., i{ 1 - “ Ax)/a Ax)} + u x+iixl £{1 +(x+ Ax)/aAx} 


i.e. 


3 

u+(At)^ + 0(At) 2 


- K i+ « _ ^){“' ( 4 * i ^ + ^S + o<ax,> } 

+ K 1 + ; + ^){" +<Ajt) l +i( 4 j ' )i S +0<Ax)i } 


Therefore 


du 


+ 0(At) 


dt 

Therefore 


= (iU w+iMfS 


«i,;ax ta)+ 2 4, Va* ! 


du J . d 2 u 

ft - frx iux,+ ^e? 


) +0 ( 4x ^r) +0<ix|! 


□ ( 8 . 2 ) 


This limiting equation is the diffusion equation for the Ornstein-Uhlenbeck 
process, which is defined to be the Gaussian process with EZ, = 0 and 
covariance function g(s,t) = ae - ^ 5- ' 1 (a > 0, P > 0). Problem 8.1 
showed a method of constructing it from a Wiener process. 


EXERCISES 

1. If {X, : t e [0, oo)} is the Wiener process with parameter A, for each 
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of the following processes {Z,: £e[0, oo)} find EZ,. covfZ,,Z ( ) (s < r), 
and state whether the process is stationary in the wide sense, and whether 
it is Gaussian. 

(i) Z, = M + X, (a > 0), (ii) Z, = X t+t -X, (t > 0), 

(iii) z , = (l (0 < t < 1), Z, = 0 (t> 1), 

(iv) Z, = Y f+l — Y t , where Y t is Gaussian with 

E Y, = a+fit, cov(y„ y t+T ) = e-»M (y > 0) 


2. Consider the diffusion limit of an unrestricted random walk on the 
real axis. If at time t, X t = x, then at time r+At 


y _ y 

( + Ar - A (n+ 1)AJ 


(n — l)Ax = x —Ax with probability p x 
nAx = x with probability r x 
(n+ l)Ax = x+Ax with probability p x 


where p x and r x are continuous functions of x, and 2 p x +r x = 1 for every 
value of x. 


Let u xt dx = pr{A" ( e (x, x + dx)}. If Ax and Af both tend to zero in 
such a way that (Ax) 2 /A/ -> ft, a constant, show that in the limit u = u xi 
satisfies 


du 8 2 

T ," "spW 


3. Consider a particle which starts at the origin and carries out a random 

walk according to Problem 8.2. If <p t (0) is the characteristic function for 
Z,, and if ij/(G) is the cf for a N(/?, a) rv, show by a decomposition of</> lx . (0) 
based on the last At that ' ' 

<t>,(9) = 

4. In a model for gas diffusion through a porous membrane there are 2 
cells, A and B, each containing 2c molecules. These molecules are of 2 
types. there are 2c black ones and 2c white ones. After each unit interval 
in time one molecule is chosen at random from each cell, and the 2 mole¬ 
cules are interchanged. The rv X n is the number of black molecules in A 
at time n. Consider the Markov chain \Z H :n = 0,1,2,...}, where 
z « — X„—c. Find difference equations in k and n for p£ n) = prfZ„ = k) 
by a decomposition based on the last step. 

Consider the diffusion process arising as follows. If at time instant t, 
Z, — z, then at t+At, Z (+Al = z—Az, z or z+Az, where the process has 
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state space (— c, c). If both Az and At tend to zero, and c tends to infinity in 
such a way that (Az) 2 /At -* y and (c At) -1 ->■ jS, show that in the limit u z , 
satisfies the Omstein-Uhlenbeck process differential equation 


where 


du 8 . d 2 u 

J, ~ 


u : t dz = pr{Z, e(z,z + dz)} 


5. Suppose that {Z ( : t e [0, oo)} is the Omstein-Uhlenbeck process. 
If Z, given that Z 0 = z is N{ze - ^', y(l — e~ 2fil )/(2P)}, show that u XI dx = 
pr{Z, 6 (x, x+dx)\Z 0 = z} satisfies equation 8.2. Note that the equilibrium 
distribution of Z, is given by letting t -* oo. and so is N(0,y/2/?), which 
does not depend on z. 


Recommendations for Further Reading 

A fairly elementary introduction to probability theory is volume 1 of 
An Introduction to Probability Theory and its Applications by William 
Feller, the third edition of which was published by John Wiley in 1968. 

General treatments of stochastic processes at about the level of this 
Problem Solver are given in The Theory of Stochastic Processes by 
David R. Cox and Hilton D. Miller, published by Methuen in 1965, and in 
The Elements of Stochastic Processes with Applications to the Natural 
Sciences by Norman T. J. Bailey, published by John Wiley in 1964. 

Some specific applications are made in Queues by David R. Cox and 
Walter L. Smith, published by Methuen in 1961, The Theory of Storage 
by P. A. P. Moran, published by Methuen in 1959, and Stochastic Models 
for Social Processes by David J. Bartholomew, the second edition of 
which was published by John Wiley in 1973. 
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